MLOps, Model Serving and Lifecycle Management

Introduction and Context

Machine-learning systems differ from conventional software because their behaviour depends not only on source code but also on data, feature transformations, model parameters, runtime infrastructure, and changing real-world conditions. A model that performs well during experimentation may fail in production because the input distribution changes, a feature pipeline is inconsistent, latency is excessive, or the model cannot be reproduced. MLOps is the set of practices, tools, and organisational processes used to develop, deploy, operate, monitor, and improve machine-learning systems reliably.

MLOps applies DevOps principles—automation, version control, continuous integration, continuous delivery, observability, and collaboration—to the additional requirements of machine learning. It also incorporates data engineering, statistics, model governance, and responsible AI. The objective is not simply to place a model behind an API. It is to create a controlled lifecycle in which every production prediction can be related to a model version, data version, code version, configuration, and deployment environment.

Cloud platforms are particularly suitable for MLOps because they provide elastic compute, managed storage, container orchestration, workflow services, model registries, monitoring systems, and identity controls. However, cloud services do not automatically create a sound MLOps process. Architecture, validation, security, cost management, and accountability remain engineering responsibilities.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain the purpose and stages of an MLOps lifecycle.

2. Distinguish experiment tracking, data versioning, model versioning, and model registration.

3. Design a reproducible training and deployment pipeline.

4. Compare online, batch, streaming, and edge inference patterns.

5. Select appropriate validation, monitoring, drift-detection, and retraining strategies.

6. Explain rollback, canary deployment, blue-green deployment, and champion–challenger evaluation.

7. Identify governance, privacy, security, ethical, and cost risks.

8. Describe how LLMOps extends MLOps for large language model applications.

1. The Machine-Learning Lifecycle

A production ML lifecycle normally contains the following connected stages:

Problem definition. The team specifies the decision to support, the prediction target, constraints, users, and success criteria. A technically accurate model is not necessarily useful if it does not improve a business or societal outcome.

Data acquisition and preparation. Data is collected, labelled, cleaned, transformed, and partitioned. Data lineage records where data came from, when it was collected, how it was transformed, and which permissions apply.

Experimentation. Researchers compare algorithms, features, hyperparameters, and preprocessing choices. Results should be recorded rather than kept only in notebooks.

Training and evaluation. A reproducible job produces a model and evaluation artefacts. Evaluation must use an appropriate test design, such as a time-based split for forecasting or a group-based split where records from the same person must not occur in both training and testing data.

Registration and approval. A candidate model is stored in a model registry with metadata, validation results, lineage, and an approval status.

Deployment. The model is exposed through an online endpoint, batch job, streaming service, embedded application, or edge device.

Monitoring and feedback. The system measures service health, prediction behaviour, data quality, model performance, fairness indicators, and cost.

Maintenance. The team investigates incidents, retrains models, promotes improved versions, or rolls back unsafe versions.

These stages are iterative rather than strictly linear. A monitoring signal may trigger data investigation, a revised feature definition, or a change in the original problem formulation.

2. Reproducibility and Experiment Tracking

An experiment is more than a model file. To reproduce it, one generally needs:

• source-code revision;
• training and validation data references;
• feature and preprocessing definitions;
• algorithm and hyperparameters;
• random seeds, where relevant;
• software dependencies and hardware information;
• evaluation metrics and plots; and
• the resulting model and other artefacts.

An experiment-tracking system records these items using runs. A run may contain parameters such as learning rate and tree depth, metrics such as validation accuracy and recall, tags such as project or owner, and artefacts such as a confusion matrix or serialized model.

A useful distinction is between reproducibility and repeatability. Repeatability means that the same team can execute the same procedure in the same environment and obtain consistent results. Reproducibility more broadly means that an independent person can recreate the result using the documented materials. Exact numerical equality may not always be possible because of nondeterministic GPU operations, distributed execution, or changing dependencies; the acceptable tolerance should therefore be specified.

Experiment tracking should not encourage selection of a model solely because it has the highest metric. The metric must be appropriate to the operational objective. For example, in fraud detection, precision, recall, investigation capacity, and expected financial loss may be more informative than accuracy alone.

3. Data, Feature, and Model Versioning

Data versioning

A dataset version should identify its contents and transformation history. Object storage can hold immutable files, while a data catalogue records schema, ownership, quality results, and access policy. A cryptographic hash can help identify whether an artefact has changed, but a hash alone does not explain semantic changes such as a revised label definition.

Data versioning should record:

• source and collection time;
• schema and units;
• filtering and labelling rules;
• personally identifiable information handling;
• sampling method;
• known missingness and quality issues; and
• the relationship to model-training runs.

Training data must be separated from evaluation data to prevent leakage. For time-dependent problems, a random split can allow future information to influence training. A chronological split is usually more appropriate.

Feature versioning

A feature is a measurable input derived from raw data. The definition of “customer activity in the previous 30 days,” for example, includes a time window, event types, timezone, and treatment of missing events. Changing any of these may change model behaviour. Feature stores can provide an offline store for training and an online store for low-latency retrieval. The important design principle is training-serving consistency: the feature calculated during training must represent the feature available at prediction time.

Model versioning and registries

A model registry is a controlled catalogue of model artefacts and their metadata. A registered model may have versions such as version 12 and version 13, but numerical ordering does not itself establish quality. Each version should include:

• model artefact and signature;
• input and output schema;
• training-data reference;
• code and environment reference;
• evaluation results and validation reports;
• intended use and limitations;
• owner and approval history; and
• deployment status.

A registry often uses lifecycle states such as development, candidate, approved, deployed, retired, and rejected. Approval should be based on explicit gates rather than informal agreement.

4. Pipelines and Automation

An ML pipeline is an automated sequence of tasks with defined inputs, outputs, dependencies, and failure behaviour. A typical training pipeline is:

1. validate incoming data;

2. create a versioned dataset;

3. engineer features;

4. train one or more candidates;

5. evaluate against a fixed test protocol;

6. check fairness, robustness, and operational constraints;

7. register the candidate; and

8. deploy only when approval conditions are satisfied.

A pipeline should be idempotent where practical: running it again with the same inputs should not create contradictory results or corrupt shared state. Tasks should be restartable, and intermediate outputs should be stored when recomputation is expensive. Dependency graphs are preferable to long scripts because independent tasks can run in parallel and failed tasks can be retried.

Continuous integration (CI) tests code, data transformations, schemas, and packaging whenever changes are submitted. Continuous delivery (CD) automatically prepares a release and may deploy it after approval. Continuous deployment releases it automatically when all controls pass. In regulated or high-risk settings, continuous delivery with human approval may be more suitable than unrestricted continuous deployment.

A strong pipeline separates:

• training code, which creates a model;
• validation code, which decides whether it meets requirements; and
• serving code, which loads the approved model and produces predictions.

This separation makes it harder for a developer to unintentionally bypass acceptance criteria.

5. Validation and Release Gates

Validation operates at several levels.

Schema validation checks column names, types, ranges, units, and null rates. Data-quality validation checks duplicates, impossible values, label consistency, and distribution anomalies. Model validation checks predictive metrics on an untouched evaluation set. Operational validation checks latency, throughput, memory use, startup time, and failure handling. Responsible-AI validation can examine subgroup performance, explainability requirements, privacy exposure, and prohibited uses.

A release gate should combine minimum thresholds and comparative criteria. For example, a candidate may be required to have recall above a specified threshold, no unacceptable subgroup disparity, and latency below the service objective. A candidate that improves average accuracy but violates a safety constraint should not be promoted.

Worked example: capacity and latency

Suppose an online endpoint must process 120 requests per second. One replica can sustain 35 requests per second under the required latency target. If capacity is divided evenly and a 20% operational reserve is required, the required capacity is

120 × 1.20 = 144 requests per second.

The number of replicas is therefore the smallest integer not less than 144/35, which is 5 replicas. This calculation assumes stable per-replica throughput, approximately even load distribution, and no bottleneck elsewhere. Load testing is still necessary because network, model-loading, queueing, and autoscaling behaviour may alter the result.

6. Model-Serving Patterns

Online inference

Online inference returns a prediction synchronously or near synchronously in response to a request. It is suitable for credit-risk decisions, recommendations, search ranking, and interactive assistants. The design must specify a latency objective, availability target, request schema, timeout, retry policy, authentication, and fallback behaviour.

A serving container commonly loads the model at startup and exposes a health endpoint and prediction endpoint. Models should generally be loaded once rather than from storage for every request. Batching several requests can increase hardware efficiency, but excessive batching increases waiting time.

Batch inference

Batch inference processes a known collection of records on a schedule. It is suitable for nightly customer scores, monthly risk reports, or catalogue recommendations. Batch systems can use larger models at lower cost because they do not need to respond to each individual request. Outputs should be versioned and linked to the input snapshot and model version.

Streaming inference

Streaming inference consumes events continuously and emits predictions as events arrive. It is useful for transaction monitoring and telemetry. It introduces requirements for event ordering, late-arriving data, duplicate handling, state management, and exactly-once or at-least-once processing semantics. The business consequence of duplicate predictions must be understood rather than assumed away.

Edge and embedded inference

An edge model runs near the data source, reducing network latency and supporting intermittent connectivity. Constraints include limited memory, energy, update mechanisms, device security, and model compatibility. Model compression or quantization may be useful, but any reduction in size must be evaluated for its effect on accuracy and fairness.

7. Deployment Strategies

A recreate deployment stops the old service and starts the new one. It is simple but can cause downtime. A rolling deployment replaces instances gradually and is efficient for compatible versions. A blue-green deployment maintains two environments: traffic is switched from blue to green after validation, and blue remains available for rapid reversal.

A canary deployment sends a small proportion of traffic to the new model and compares it with the existing version. Traffic can increase gradually if latency, errors, business outcomes, and safety metrics remain acceptable. Canary analysis is stronger when users or cases are assigned consistently, because random reassignment can make comparison noisy.

A shadow deployment sends copies of production requests to a candidate without using its predictions for decisions. It is valuable for measuring latency and prediction differences without exposing users to the candidate’s actions. A champion–challenger arrangement compares the current production model, the champion, with one or more challengers under a controlled evaluation design.

Rollback means returning to a previously approved version. A rollback plan must include model artefacts, serving images, feature definitions, schemas, configuration, and database compatibility. Reverting only the model may fail if the feature pipeline has also changed.

8. Monitoring and Observability

Monitoring should cover four broad dimensions.

System monitoring includes availability, CPU and memory use, accelerator utilisation, queue length, throughput, timeout rate, and latency percentiles. The 95th or 99th percentile can reveal slow requests that an average conceals.

Data monitoring includes missing values, schema changes, range violations, category frequencies, and input distributions. It can identify upstream failures before model metrics become available.

Prediction monitoring includes prediction distributions, confidence or score distributions, rejection rates, and changes in decision proportions.

Performance monitoring uses ground-truth outcomes when they become available. Depending on the task, this may include precision, recall, calibration, mean absolute error, or business loss. Ground truth is often delayed, incomplete, or affected by the model’s own decisions, so interpretation requires care.

Monitoring must produce actionable alerts. An alert should identify the affected service, severity, evidence, owner, and response procedure. Excessive alerts cause fatigue; insufficient alerts allow silent degradation. Logs should avoid unnecessary personal data and should include correlation identifiers so that a prediction can be traced without exposing sensitive content.

9. Drift Detection and Retraining

Data drift is a change in the distribution of inputs. Concept drift is a change in the relationship between inputs and the target. Label drift is a change in the target or outcome distribution. These categories are related but not interchangeable. A change in input distribution does not necessarily reduce predictive performance, and stable inputs do not guarantee a stable input–outcome relationship.

For a numerical feature, a team may compare a reference window with a current window using quantiles, histograms, or a statistical distance. For categorical data, frequency changes can be monitored. For a discrete distribution, the population stability index is sometimes used:

PSI = Σᵢ (pᵢ − qᵢ) ln(pᵢ/qᵢ),

where pᵢ is the reference proportion in bin i and qᵢ is the current proportion. The bins must be defined consistently, and neither proportion should be zero without an appropriate smoothing policy. PSI is a screening indicator, not proof that a model has degraded; thresholds are domain-dependent and should be validated empirically.

If labels are available, direct performance monitoring is usually more meaningful than input drift alone. Retraining may be scheduled periodically, triggered by performance deterioration, triggered by significant data change, or initiated after a business or regulatory change. Automatic retraining should not automatically imply automatic deployment. A newly trained model must pass the same validation and approval controls as any other release.

10. Governance, Security, and Responsible Operation

MLOps creates an audit trail, but governance requires more than logs. Organisations should define ownership, permitted use, retention periods, access roles, incident procedures, and model-risk classifications. Privacy principles include data minimisation, purpose limitation, appropriate retention, encryption, and controlled access. Training data may contain sensitive information even when the model does not explicitly expose it.

Security threats include poisoned training data, compromised dependencies, stolen model files, adversarial inputs, prompt injection in LLM systems, insecure model deserialization, and excessive API access. Controls include dependency scanning, signed artefacts, isolated execution, secrets management, network restrictions, rate limits, input validation, and least-privilege identity policies.

Fairness assessment should consider the relevant protected or vulnerable groups, the decision context, and the consequences of errors. There is no universally correct fairness metric; different metrics can conflict, particularly when group base rates differ. A responsible process documents the choice, limitations, remediation actions, and residual risk.

Regulatory obligations vary by jurisdiction and application. High-impact uses may require documentation, human oversight, transparency, impact assessment, records of training data and testing, and mechanisms for contesting decisions. A model card or system card can communicate intended use, evaluation conditions, limitations, and risks.

11. LLMOps

LLMOps applies lifecycle practices to applications using large language models. In addition to conventional model versioning, it must manage prompts, system instructions, retrieval indexes, embedding models, document versions, tool definitions, safety filters, and evaluation datasets.

Important LLMOps practices include:

• versioning prompts and retrieval configurations;
• evaluating factuality, relevance, toxicity, refusal behaviour, and task success;
• testing against adversarial and prompt-injection inputs;
• monitoring token usage, latency, cost, and provider changes;
• protecting confidential context in logs and traces;
• grounding responses in approved sources where appropriate; and
• retaining human review for high-impact outputs.

Retrieval-augmented generation (RAG) introduces a second lifecycle: source documents must be governed, chunking and embedding must be reproducible, indexes must be refreshed, and retrieved evidence should be traceable. Fine-tuned models require training-data provenance and careful evaluation for memorisation and unintended behaviour. Because LLM outputs are probabilistic and may appear authoritative while being wrong, a fluent response is not a sufficient quality criterion.

12. Illustrative Case: Cloud Fraud Detection

Background

A fictional online retailer uses an ML model to identify potentially fraudulent card transactions. The existing rules engine produces many false positives, delaying legitimate orders. The retailer wants a cloud-based model that scores transactions within 150 milliseconds and supports investigator review.

Problem and analysis

The team discovers that fraud labels arrive several weeks after a transaction. A random training split would allow transactions from the same customer and future periods to leak into evaluation. The team therefore uses a time-based split, trains on earlier months, validates on a later month, and reserves the most recent month for final testing. It measures recall at the investigator team’s fixed review capacity, precision, latency, and performance across relevant customer groups.

Approach

Raw transactions are stored in an immutable, access-controlled object store. A data pipeline validates schema, removes unauthorised fields, and creates a versioned feature set. Online features are calculated using the same definitions as offline features, with recent aggregates stored in a low-latency feature store. Each training run records code revision, data snapshot, features, hyperparameters, metrics, and container image.

The candidate is registered only if it meets minimum recall, precision, latency, and subgroup-performance requirements. It is first deployed in shadow mode. After two weeks, the team uses a canary release for 10% of traffic. The old model remains the champion. Alerts cover API latency, input missingness, score distribution, investigator workload, and confirmed fraud performance when labels arrive.

Outcome

The candidate reduces false positives while maintaining the required recall. During the canary, a payment-provider schema change causes a sudden increase in missing features. The data-quality alert stops further traffic expansion, and the team rolls back to the champion while correcting the upstream integration. No unvalidated model becomes the sole production decision-maker.

Learning points

The case demonstrates that MLOps is a socio-technical process. Version control, time-aware evaluation, feature consistency, staged deployment, monitoring, and rollback work together. A high offline score alone would not have detected the production schema failure or guaranteed acceptable investigator workload.

13. Challenges and Best Practices

Common challenges include fragmented ownership, undocumented notebooks, inconsistent environments, training-serving skew, delayed labels, excessive cloud cost, and pressure to deploy before validation. Small teams should avoid adopting every available tool. A minimal but coherent platform—versioned storage, experiment tracking, automated tests, a registry, reproducible pipelines, monitoring, and documented approval—often provides more value than a complex platform that nobody maintains.

Recommended practices are:

1. Begin with a clearly defined decision and operational success measure.

2. Make datasets, features, code, environments, and models traceable.

3. Prefer immutable artefacts and explicit metadata.

4. Test data transformations as seriously as application code.

5. Use time- or group-aware evaluation when random splitting is invalid.

6. Separate candidate creation from candidate approval.

7. Test production-like latency, load, failure, and security conditions.

8. Release gradually and maintain a tested rollback path.

9. Monitor data, predictions, outcomes, system health, and cost.

10. Treat retraining as a controlled change, not an automatic cure.

11. Minimise sensitive data in logs and restrict access by role.

12. Document intended use, limitations, residual risks, and human responsibilities.

Chapter Summary

MLOps provides the lifecycle discipline required to operate machine-learning systems reliably. It connects problem definition, data and feature management, experiment tracking, reproducible pipelines, validation, model registries, deployment, monitoring, drift detection, retraining, and rollback. Online, batch, streaming, and edge inference serve different latency, scale, and connectivity requirements. Deployment methods such as canary, blue-green, shadow, and champion–challenger releases reduce operational risk. Monitoring must include infrastructure, data, predictions, performance, responsible-AI indicators, and cost. Drift signals support investigation but do not by themselves prove model failure. LLMOps extends these principles to prompts, retrieval data, tools, safety evaluation, token costs, and probabilistic generation. Effective MLOps is therefore both a technical and governance capability.

Key Terms

MLOps: Practices for developing, deploying, monitoring, and maintaining ML systems.

Experiment tracking: Recording parameters, metrics, code, data references, and artefacts for experiments.

Data lineage: The documented origin and transformation history of data.

Data versioning: Maintaining identifiable versions of datasets and their metadata.

Feature store: A system that manages reusable features for training and inference.

Training-serving skew: A mismatch between features used during training and those available in production.

Model registry: A controlled catalogue of model versions, metadata, and approval states.

Pipeline: An automated dependency-aware sequence of data, training, validation, or deployment tasks.

Model signature: A description of a model’s expected input and output schema.

Online inference: Producing a prediction in response to an individual request.

Batch inference: Producing predictions for a collection of records on a schedule or job.

Canary deployment: Gradually exposing a new version to a small share of traffic.

Shadow deployment: Running a candidate on copied traffic without using its outputs for decisions.

Rollback: Returning a production service to a previously approved version.

Data drift: Change in the distribution of model inputs.

Concept drift: Change in the relationship between inputs and the target outcome.

Observability: The ability to understand system state through logs, metrics, and traces.

LLMOps: Lifecycle and operational practices for applications using large language models.

Model card: Documentation describing a model’s uses, evaluation, limitations, and risks.

Review Questions

Short Answer Questions

1. Why is experiment tracking important in an ML project?

2. Distinguish data drift from concept drift.

3. What is the purpose of a model registry?

4. State one advantage and one limitation of batch inference.

5. Why should a rollback plan include feature definitions as well as model files?

Long Answer Questions

1. Explain the stages of an MLOps lifecycle and show how they form an iterative process.

2. Compare online, batch, streaming, and edge inference, including suitable use cases and engineering constraints.

3. Describe how a training pipeline can enforce reproducibility and release quality.

4. Explain canary, blue-green, shadow, and champion–challenger deployment strategies.

5. Discuss monitoring, drift detection, retraining, and governance as connected activities.

Analytical / Application Questions

1. Design an MLOps architecture for a medical appointment no-show model. Identify data controls, validation gates, deployment pattern, monitoring signals, and rollback conditions.

2. An online service receives 200 requests per second. Each replica sustains 50 requests per second under the required latency, and the team requires a 25% capacity reserve. Calculate the minimum number of replicas and state the assumptions behind the calculation.

3. A candidate model improves aggregate accuracy but increases error rates for a vulnerable subgroup and has higher 99th-percentile latency. Decide whether it should be deployed and justify the decision using technical, ethical, and governance criteria.

Multiple Choice Questions with Answers

Question 1

Which artefact most directly supports tracing a production model back to its training process?

A. A model filename containing a date

B. A lineage record linking code, data, features, environment, and metrics

C. A dashboard showing CPU utilisation

D. A list of API endpoints

Correct Answer: B. A lineage record linking code, data, features, environment, and metrics

Question 2

Which inference pattern is generally most suitable for nightly customer scoring?

A. Batch inference

B. Synchronous online inference

C. Edge inference only

D. Shadow inference

Correct Answer: A. Batch inference

Question 3

What is the main purpose of a canary deployment?

A. To remove all monitoring requirements

B. To expose a new version gradually to a limited portion of traffic

C. To train a model without labelled data

D. To guarantee zero latency

Correct Answer: B. To expose a new version gradually to a limited portion of traffic

Question 4

Which situation is an example of training-serving skew?

A. The training and serving systems use the same feature definition

B. A model is evaluated on a held-out test set

C. A feature uses a 30-day window in training but a 7-day window in production

D. A model is stored in a registry

Correct Answer: C. A feature uses a 30-day window in training but a 7-day window in production

Question 5

Why is a time-based split often preferred for a forecasting problem?

A. It guarantees perfect accuracy

B. It prevents future information from leaking into training

C. It removes the need for monitoring

D. It always produces more training records

Correct Answer: B. It prevents future information from leaking into training

Question 6

What does a shadow deployment do with a candidate model’s predictions?

A. It immediately replaces the champion’s decisions

B. It discards the production model

C. It evaluates the candidate on copied traffic without using its outputs for decisions

D. It retrains the candidate after every request

Correct Answer: C. It evaluates the candidate on copied traffic without using its outputs for decisions

Question 7

A service needs 120 requests per second, each replica handles 35 requests per second, and a 20% reserve is required. What is the minimum number of replicas?

A. 3

B. 4

C. 5

D. 6

Correct Answer: C. 5

Question 8

Which statement about data drift is correct?

A. It always proves that predictive performance has declined

B. It is a change in the distribution of model inputs

C. It refers only to software dependency changes

D. It can occur only when labels are available

Correct Answer: B. It is a change in the distribution of model inputs

Question 9

Which is a distinctive LLMOps concern?

A. Versioning prompts and retrieval indexes

B. Eliminating all model evaluation

C. Using only CPU monitoring

D. Avoiding document provenance

Correct Answer: A. Versioning prompts and retrieval indexes

Question 10

What is the safest interpretation of automatic retraining?

A. Every retrained model should be deployed immediately

B. Retraining removes the need for approval gates

C. Retraining creates a candidate that should pass validation and approval

D. Retraining guarantees that drift has been corrected

Correct Answer: C. Retraining creates a candidate that should pass validation and approval