Deployment, MLOps and Model Monitoring

Introduction

A machine-learning model has limited practical value until it can operate reliably in the environment where decisions are made. Training usually occurs on curated historical data, using substantial computational resources and careful experimentation. Deployment places the resulting model in contact with real users, business processes, devices, and continuously changing data. This transition introduces challenges that are not solved by improving predictive accuracy alone.

A deployed model must be packaged, served, monitored, secured, updated, and eventually retired. Its performance may decline because the input population changes, the relationship between inputs and outcomes changes, the software environment changes, or the surrounding business process changes. Machine learning operations, commonly called MLOps, provides practices for managing these issues throughout the model life cycle.

This chapter examines batch and online inference, containers, model registries, cloud and edge deployment, CI/CD, monitoring, drift, retraining, scalability, governance, and technical debt.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish training, inference, batch serving, and online serving;
• design a basic model packaging and deployment workflow;
• explain the roles of containers, model registries, CI/CD, and feature pipelines;
• compare cloud, edge, and hybrid deployment;
• select appropriate service and system-monitoring metrics;
• distinguish data drift, concept drift, and model-performance degradation;
• design a monitoring and retraining policy;
• analyze scalability, reliability, security, fairness, and technical debt in ML systems.

1. From a Trained Model to a Production System

A model is an artifact that maps input features to predictions. A production ML system is broader. It includes data ingestion, feature transformation, the model, prediction logic, validation, logging, monitoring, access control, and operational procedures.

A useful conceptual pipeline is:

Data source → validation → feature computation → model inference → post-processing → decision or recommendation → logging and monitoring

The model may be represented as a serialized file, but that file alone is insufficient. The exact preprocessing steps, feature definitions, label interpretation, dependency versions, and model configuration must also be preserved. If a model was trained after standardizing a variable, production must apply the same transformation using the training parameters. Reimplementing preprocessing independently can create training-serving skew.

Training and inference

Training estimates model parameters from historical data. Inference applies fixed parameters to new observations. Training is generally less frequent and computationally expensive; inference may occur millions of times per day and must meet operational constraints such as latency and availability.

A model can be accurate in a test notebook and still fail in production because:

• production data contains missing or unexpected values;
• feature definitions differ from those used during training;
• the service cannot meet latency requirements;
• dependencies are incompatible;
• the model has no rollback mechanism;
• predictions are not monitored;
• the model’s output is not integrated into the decision process.

Therefore, deployment should be treated as an engineered life-cycle process rather than a final upload step.

2. Batch and Online Inference

Batch inference

In batch inference, predictions are generated for a collection of records at scheduled intervals. Examples include producing a daily customer-churn list, scoring all transactions overnight, or generating weekly demand forecasts.

Batch systems are appropriate when immediate responses are unnecessary. They usually offer simpler operations, efficient bulk computation, and easier recovery. A failed job can often be rerun for a defined data interval. Their disadvantages include prediction delay and the need to manage duplicate or missing batches.

A batch workflow should record:

• the data snapshot or time window used;
• the model version;
• the feature-processing version;
• job start and completion times;
• record counts and validation results;
• output location and downstream consumers.

Online inference

Online, or real-time, inference returns a prediction in response to an individual request. Examples include fraud screening during payment authorization, ranking search results, and estimating delivery time in a mobile application.

An online service normally exposes an API endpoint. A request may contain raw attributes or references to features stored elsewhere. The service validates the request, computes or retrieves features, invokes the model, applies business rules, and returns a response.

Important service properties include latency, throughput, availability, reproducibility, and graceful failure. A fallback may be a rules-based decision, a previous model, or a conservative default. Fallbacks must be designed carefully because they can introduce systematic unfairness or safety risks.

Choosing between them

The choice depends on decision latency, data freshness, volume, cost, and operational risk. Hybrid systems are common: a daily batch job computes expensive customer-level features, while an online service combines those features with request-time information.

3. Packaging and Serving Models

A reliable deployment package should include the model artifact, inference code, preprocessing logic, dependency specifications, configuration, health checks, and documentation. The package should be immutable: changing the contents without changing its version makes diagnosis and rollback difficult.

Containers

A container bundles application code and its dependencies into a reproducible runtime environment. It does not normally include a complete independent operating-system kernel; instead, containers share the host kernel while isolating processes and filesystems. This distinguishes containers from virtual machines, which generally virtualize a complete operating-system environment.

A typical model-serving container contains:

1. a base runtime such as a supported Python or Java environment;

2. pinned library versions;

3. model and preprocessing artifacts;

4. an API server or batch entry point;

5. health and readiness checks;

6. logging configuration;

7. a non-privileged execution user where possible.

Containers improve reproducibility, but they do not automatically guarantee security or correctness. Images must be scanned for vulnerabilities, rebuilt when dependencies require security updates, and deployed with least privilege.

APIs and contracts

An API contract specifies request fields, types, permitted ranges, missing-value behavior, response format, error codes, and versioning rules. Schema validation should occur before inference. A service should distinguish client errors, such as invalid input, from server errors, such as unavailable dependencies.

A prediction response may include a model version and a timestamp. Exposing confidence scores requires care: a model probability is not necessarily a calibrated probability, and users should not interpret it as certainty without validation.

Model registries

A model registry stores and organizes model versions together with metadata. Useful metadata includes:

• training-data identifiers and time ranges;
• evaluation metrics and validation population;
• feature and code versions;
• approval status;
• intended use and known limitations;
• responsible owner;
• deployment history.

A registry supports promotion through stages such as development, validation, staging, and production. It should not be treated merely as a file store; its value lies in traceability, approval control, and reproducibility.

4. Cloud, Edge, and Hybrid Deployment

Cloud deployment

Cloud platforms provide elastic compute, managed storage, networking, orchestration, monitoring, and specialized hardware. They are useful when workloads vary, teams require rapid experimentation, or models need centralized management. However, cloud deployment introduces recurring costs, dependency on network connectivity, data-transfer concerns, and possible regulatory restrictions on data location.

Scaling can be vertical, by increasing the resources of a machine, or horizontal, by adding service instances. Horizontal scaling generally improves capacity and resilience, but it requires stateless service design, load balancing, consistent model versions, and careful management of shared resources.

Edge deployment

Edge inference runs near the data source, such as on a mobile device, industrial controller, vehicle, or camera. It can reduce network latency, preserve privacy by keeping raw data local, and continue operating during connectivity interruptions. Constraints include limited memory, energy, computing power, and update capability.

Model compression techniques may include quantization, pruning, and knowledge distillation. These techniques can reduce resource requirements but may reduce accuracy or alter behavior. Any compressed model must be evaluated on representative data under the actual device constraints.

Hybrid deployment

Hybrid architectures divide computation between edge and cloud. An edge device might perform immediate detection while sending summarized events to the cloud for analysis and retraining. Such systems require explicit synchronization, secure update mechanisms, and policies for inconsistent model versions.

Deployment choice should consider latency, privacy, availability, cost, hardware constraints, data sovereignty, and update frequency rather than selecting a platform solely because it is popular.

5. CI/CD and the ML Life Cycle

Continuous integration (CI) automatically tests changes to code and configuration. Continuous delivery or deployment (CD) automates the process of making validated software available in a target environment. In ML systems, the pipeline must test not only software but also data, features, models, and operational behavior.

A practical ML delivery pipeline may contain the following stages:

1. Source control: version code, configuration, schemas, and infrastructure definitions.

2. Data validation: check schema, ranges, missingness, duplicates, and volume changes.

3. Feature tests: verify transformations and ensure training and serving implementations agree.

4. Model training or retrieval: produce a versioned artifact using controlled data.

5. Evaluation: assess predictive performance, subgroup behavior, calibration where relevant, and resource requirements.

6. Packaging: build a tested container or batch package.

7. Staging: test with production-like traffic and dependencies.

8. Release: use controlled promotion, such as canary or shadow deployment.

9. Monitoring: observe system behavior and prediction quality.

10. Rollback or retraining: respond to failures according to documented policies.

A canary release sends a small proportion of traffic to a new version before full promotion. A shadow deployment sends copied requests to a new version but does not use its decisions. Shadowing is useful for comparing predictions without immediately changing outcomes.

Automated tests should include unit tests, integration tests, data-quality tests, load tests, security tests, and regression tests. A model that improves average accuracy but violates latency or fairness requirements should not automatically be released.

6. Observability and Monitoring

Observability is the ability to understand system behavior from its outputs, logs, metrics, and traces. Monitoring is the planned collection and interpretation of these signals.

System monitoring

Common infrastructure and service metrics include:

• request rate and throughput;
• latency percentiles, especially p95 and p99;
• error and timeout rates;
• CPU, memory, accelerator, and disk utilization;
• queue length;
• service availability;
• feature-store and database response times.

The arithmetic mean latency can hide a poor user experience when a small fraction of requests is very slow. Percentiles describe thresholds below which a specified fraction of observations falls. For example, a p95 latency of 200 milliseconds means approximately 95% of measured requests are at or below 200 milliseconds, subject to the measurement method and sample quality.

Data-quality monitoring

Data checks should examine schema, type validity, missing values, ranges, category frequencies, duplicate rates, freshness, and unexpected distributions. Monitoring only infrastructure cannot detect that a critical feature has become constant or that a new category has appeared.

Prediction monitoring

Track prediction distributions, confidence or score distributions, rejection rates, and decision frequencies. Abrupt changes may indicate upstream failures, population changes, or model problems. Prediction monitoring does not prove that predictions are correct, because labels may arrive later.

Performance monitoring

When ground-truth outcomes become available, calculate suitable evaluation metrics. Classification metrics may include precision, recall, false-positive rate, and calibration. Regression metrics may include mean absolute error or root mean squared error. Metrics must match the decision context; a single aggregate score can conceal poor performance for an important subgroup.

Monitoring dashboards should include thresholds, owners, alert severity, and runbooks. An alert without a response procedure creates noise rather than operational control.

7. Drift and Model Decay

Drift describes a change affecting data, predictions, or the relationship between inputs and outcomes.

Data drift

Data drift is a change in the distribution of input variables. For example, the age distribution of applicants may change after a new marketing campaign. Data drift does not necessarily imply worse model performance, particularly if the model remains valid for the new population.

Concept drift

Concept drift occurs when the relationship between features and the target changes. A fraud pattern may change because attackers adopt new strategies. The same input pattern can then have a different outcome than it did during training.

Label or target drift

Target drift refers to a change in the distribution of the outcome variable. It can affect decision thresholds and may be caused by changes in the population or the process generating labels.

Drift detection methods include descriptive statistics, distribution comparisons, control charts, and statistical tests. Statistical significance alone is insufficient: with very large samples, negligible changes may be statistically significant. Practical impact, data quality, and downstream performance must also be considered.

A drift alert should not automatically trigger retraining. First investigate whether the change is a data pipeline error, a temporary event, a legitimate population change, or a genuine deterioration in performance.

8. Retraining and Model Life-Cycle Policies

Retraining strategies include:

• scheduled retraining, such as monthly or quarterly;
• event-triggered retraining, activated by drift or performance thresholds;
• continuous or frequent retraining, used when data and infrastructure support it;
• manual retraining, requiring expert review for high-risk applications.

Retraining introduces risks. New data may contain biased labels, leakage, corrupted records, or a temporary abnormal period. A replacement model should be compared with the incumbent on a fixed evaluation set, recent data, relevant subgroups, and operational tests.

A robust policy defines:

• the data window and inclusion rules;
• minimum performance and fairness criteria;
• approval responsibilities;
• validation and rollback procedures;
• monitoring after release;
• model retirement conditions.

Labels may be delayed. For example, loan default may be observable only months after approval. Performance monitoring must therefore account for label latency and avoid drawing premature conclusions from incomplete outcomes.

9. Scalability, Reliability, and Cost

Scalability concerns the ability to handle increased load while preserving required service properties. Capacity planning should consider request rate, computational cost per prediction, concurrency, memory requirements, and peak rather than average demand.

A simple capacity estimate can be expressed as:

Required instances ≈ peak request rate ÷ sustainable request rate per instance

This is an operational estimate, not a complete sizing equation. It assumes comparable requests and ignores redundancy, warm-up time, batching effects, and safety margins. For example, if peak demand is 120 requests per second and one instance sustainably handles 40 requests per second, at least 3 instances are needed before adding redundancy or headroom.

Reliability practices include timeouts, retries with backoff, circuit breakers, health checks, redundant instances, graceful degradation, and tested backups. Retries must be bounded: uncontrolled retries can amplify an outage. Idempotency is important for operations that might be repeated.

Cost includes compute, storage, data transfer, monitoring, engineering labor, incident response, and opportunity cost. A smaller model may be preferable when its marginal accuracy improvement does not justify substantially higher latency or cost.

10. Technical Debt in ML Systems

Technical debt is the future cost created by expedient choices. ML systems have ordinary software debt and distinctive ML debt.

Examples include:

• undocumented features and hidden dependencies;
• duplicated preprocessing code;
• weak data lineage;
• stale models and obsolete dashboards;
• untested training pipelines;
• untracked experiments;
• unclear ownership;
• feedback loops in which model decisions influence future training labels;
• configuration that exists only manually in production.

Debt is not eliminated by adopting a tool. It is managed through documentation, ownership, automated tests, reproducible pipelines, regular reviews, and retirement of unused models. Teams should allocate capacity for maintenance rather than treating it as work that can always be postponed.

11. Security, Ethics, and Governance

Production ML systems process data and influence people, so governance is part of engineering quality. Important controls include authentication, authorization, encryption in transit and at rest, secrets management, audit logs, vulnerability scanning, and secure model-update procedures.

Privacy risks include unnecessary collection, re-identification, excessive retention, and inappropriate secondary use. Data minimization and retention policies should be defined before deployment. Sensitive attributes may be needed for fairness evaluation, but their use must follow applicable law and institutional policy.

Fairness cannot be guaranteed by monitoring only overall accuracy. Teams should examine performance and error rates across relevant groups, document trade-offs, and assess whether the decision itself is appropriate. A technically accurate model can still cause harm if it is used outside its validated context.

Governance documentation should state intended use, prohibited use, assumptions, limitations, evaluation data, known failure modes, and human-oversight requirements. High-impact systems may require formal impact assessments, explainability procedures, appeal mechanisms, and regulatory review. Legal obligations vary by jurisdiction; technical teams should consult appropriate legal and compliance specialists.

12. Illustrative Case: Predictive Maintenance for a Manufacturing Line

Background

A manufacturer operates machines whose unexpected failures interrupt production. It develops a model that predicts whether a machine will require maintenance within the next seven days using temperature, vibration, pressure, and operating-cycle features.

Problem

The initial model performs well in offline evaluation. After deployment, maintenance engineers report that alerts are often late, and the service occasionally stops responding during shift changes. Investigation shows that the training data came from one factory, while the deployed system serves three factories with different sensors and operating conditions.

Analysis

The team identifies several issues:

• sensor units use different scales and calibration procedures;
• one vibration sensor sometimes repeats its last value;
• the online feature calculation differs from the training pipeline;
• the API has no explicit timeout for the feature database;
• labels arrive only after maintenance records are completed;
• the model registry does not record the sensor-calibration version.

The problem is therefore not simply model accuracy. It involves data quality, training-serving skew, delayed labels, observability, and metadata.

Approach and solution

The team first defines a versioned feature contract and validates units, ranges, freshness, and repeated-value rates. It moves shared transformations into a common tested component. The service is containerized, and the model registry records the training data, sensor schema, code version, and approval status.

The deployment uses a canary release at one factory. The team monitors API p95 latency, timeouts, missing-feature rates, alert volume, and machine-level outcomes. A fallback procedure sends high-risk cases to a conservative rules-based inspection queue when the feature service is unavailable.

Because labels are delayed, the team separates immediate data-quality alerts from later performance evaluation. After sufficient outcomes accumulate, it compares recall and false-alert rates across factories. Retraining is scheduled monthly but can be initiated earlier if validated drift and performance thresholds are reached.

Outcome

The revised system reduces feature failures and makes latency problems visible before they affect all factories. The model is not automatically retrained whenever a sensor distribution changes; instead, engineers investigate calibration changes and evaluate whether the model remains valid. This avoids treating every drift signal as evidence that new training is required.

Learning points

1. Deployment quality depends on the whole system, not only the model.

2. Feature contracts and lineage are essential for diagnosing failures.

3. Delayed labels require a monitoring design that distinguishes immediate signals from eventual performance.

4. Canary releases and fallbacks reduce operational risk.

5. Drift alerts require investigation and contextual judgment.

13. Contemporary Developments and Best Practices

Modern platforms increasingly support automated feature stores, managed model endpoints, workflow orchestration, experiment tracking, model cards, infrastructure as code, and policy-based deployment. Large-model and generative-AI systems add concerns such as prompt and retrieval monitoring, unsafe outputs, evaluation of changing external knowledge, and token or inference-cost control. These developments do not replace fundamental MLOps principles: versioning, validation, observability, security, human accountability, and controlled change.

Recommended practices include:

• begin monitoring design before deployment;
• version data, code, features, models, and infrastructure;
• use the same tested transformations in training and serving;
• define service-level objectives and model-quality thresholds;
• separate data-quality, system, and performance alerts;
• test rollback and disaster recovery rather than merely documenting them;
• evaluate important subgroups and relevant failure modes;
• use staged releases for consequential models;
• assign clear ownership and maintain runbooks;
• retire models and pipelines that no longer have a justified purpose.

Chapter Summary

Deployment converts a trained model into a functioning service or scheduled workflow. Batch inference is suitable for periodic, high-volume scoring, while online inference supports immediate decisions; hybrid systems combine both. Reliable deployment requires reproducible packaging, containers, API contracts, model registries, version control, and automated testing.

Cloud platforms provide elasticity, edge systems provide local and low-latency inference, and hybrid architectures balance these advantages. CI/CD for ML must test code, data, features, models, security, and operations. Observability combines system, data, prediction, and eventual performance monitoring.

Data drift changes input distributions, concept drift changes the input–outcome relationship, and model decay refers to declining usefulness. Drift signals should lead to investigation rather than automatic retraining. Scalable systems require capacity planning, resilience, controlled retries, and cost awareness. Security, privacy, fairness, documentation, human oversight, and regulatory compliance are essential in responsible production ML. Finally, technical debt must be managed through reproducibility, testing, ownership, and deliberate maintenance.

Key Terms

Batch inference: Generating predictions for a collection of records at scheduled times.

Online inference: Producing a prediction in response to an individual request.

MLOps: Practices for developing, deploying, operating, monitoring, and updating ML systems.

Container: An isolated runtime package containing application code and dependencies.

Model registry: A system for storing model versions and their metadata, status, and lineage.

API contract: The defined structure, validation rules, and behavior of a service interface.

Canary release: Gradual release of a new version to a small portion of traffic.

Shadow deployment: Testing a model with copied traffic without using its predictions for decisions.

Observability: The ability to understand system behavior through logs, metrics, traces, and outputs.

Training-serving skew: A mismatch between transformations or data used during training and serving.

Data drift: Change in the distribution of model inputs.

Concept drift: Change in the relationship between inputs and the target outcome.

Target drift: Change in the distribution of the outcome variable.

Feature contract: An explicit specification of feature definitions, types, units, ranges, and freshness.

Fallback: An alternative behavior used when the primary model or dependency fails.

Technical debt: Future maintenance cost caused by expedient or poorly documented choices.

Model decay: Reduction in model usefulness or predictive performance over time.

Retraining: Re-estimating model parameters using new or expanded data.

Review Questions

Short Answer Questions

1. What is the difference between batch and online inference?

2. Why is a model registry useful in a production ML workflow?

3. Define training-serving skew.

4. What is concept drift?

5. Name three categories of monitoring used in deployed ML systems.

Long Answer Questions

1. Explain the stages of an ML CI/CD pipeline and describe why ordinary software tests are insufficient by themselves.

2. Compare cloud, edge, and hybrid deployment using latency, privacy, scalability, cost, and update constraints.

3. Explain how containers, API contracts, and model registries contribute to reproducible deployment.

4. Discuss a responsible retraining policy, including delayed labels, validation, approval, and rollback.

5. Analyze the major forms of technical debt in ML systems and propose methods for controlling them.

Analytical / Application Questions

1. A prediction service receives 120 requests per second at peak demand, while one instance sustainably handles 40 requests per second. Estimate the minimum number of instances before adding redundancy and safety margin. What additional information would be needed for production sizing?

2. A fraud model shows a large change in an input distribution, but confirmed fraud rates and recall remain stable. What explanations are possible, and what investigations should precede retraining?

3. Design a monitoring and release plan for a medical-risk model whose labels become available six months after predictions. Include immediate system and data checks, delayed performance evaluation, staged deployment, and governance controls.

Multiple Choice Questions with Answers

Question 1

Which workload is most naturally suited to batch inference?

A. Fraud authorization requiring a response in milliseconds

B. Generating a daily list of customers likely to churn

C. Ranking results for an active search request

D. Detecting an obstacle during vehicle operation

Correct Answer: B. Generating a daily list of customers likely to churn

Question 2

What is the primary purpose of a model registry?

A. To replace all model evaluation with manual judgment

B. To store model versions and associated lineage and approval metadata

C. To provide network connectivity to edge devices

D. To guarantee that a model will never experience drift

Correct Answer: B. To store model versions and associated lineage and approval metadata

Question 3

Which statement correctly describes a canary release?

A. The new model receives a small portion of production traffic before wider release

B. The new model is trained only on a small dataset

C. The old model is permanently deleted before testing

D. The model runs only on an edge device

Correct Answer: A. The new model receives a small portion of production traffic before wider release

Question 4

A change in the distribution of input features is called:

A. Concept drift

B. Data drift

C. Code compilation

D. Model serialization

Correct Answer: B. Data drift

Question 5

If peak demand is 120 requests per second and one instance handles 40 requests per second, how many instances are required before adding safety margin?

A. 2

B. 3

C. 4

D. 40

Correct Answer: B. 3

Question 6

Why can p95 latency be more informative than average latency?

A. It shows the response time threshold met by approximately 95% of requests

B. It measures predictive accuracy directly

C. It eliminates the need for monitoring errors

D. It always equals the maximum latency

Correct Answer: A. It shows the response time threshold met by approximately 95% of requests

Question 7

What is training-serving skew?

A. A difference between cloud and edge hardware prices

B. A mismatch between transformations or data used in training and serving

C. A deliberate increase in model complexity

D. A method for encrypting model files

Correct Answer: B. A mismatch between transformations or data used in training and serving

Question 8

Why should drift detection not automatically trigger retraining in every case?

A. Drift can result from temporary events or data-pipeline errors rather than genuine model deterioration

B. Retraining is never useful after deployment

C. Drift can only occur in labels

D. Retraining cannot use recent data

Correct Answer: A. Drift can result from temporary events or data-pipeline errors rather than genuine model deterioration

Question 9

Which deployment approach runs a new model on copied requests without using its outputs for decisions?

A. Canary deployment

B. Shadow deployment

C. Batch deployment

D. Rollback deployment

Correct Answer: B. Shadow deployment

Question 10

Which is an example of model-specific technical debt?

A. A broken office printer

B. Duplicated preprocessing code that can diverge between training and serving

C. A larger monitor

D. A scheduled staff meeting

Correct Answer: B. Duplicated preprocessing code that can diverge between training and serving