11. MLOps, DataOps and Model Lifecycle Management

Introduction and Context

A machine-learning model is not a finished product when its training accuracy is high. In a production environment, the model must receive reliable data, operate within performance and latency constraints, interact correctly with software systems, remain explainable and secure, and continue to provide useful predictions as circumstances change. These requirements make machine learning fundamentally different from a one-time analytical exercise. A model is an operational system whose behaviour must be engineered, observed, governed and periodically revised.

MLOps addresses this challenge by applying principles of software engineering, DevOps, data engineering and machine learning to the complete model lifecycle. DataOps complements MLOps by improving the reliability, traceability and speed of data pipelines. Model lifecycle management provides the broader governance structure through which models are developed, validated, deployed, monitored, retrained, retired and audited.

The importance of these practices is especially clear in domains such as credit, healthcare, retail, manufacturing, logistics and cybersecurity. A recommendation model may become less relevant as customer preferences change. A fraud model may encounter new attack strategies. A demand-forecasting model may fail when supply conditions or consumer behaviour shift. Without automated testing, experiment tracking, monitoring and controlled deployment, such deterioration may remain hidden until it causes financial, operational or social harm.

This chapter examines the architecture and practice of reliable ML delivery. It considers CI/CD/CT pipelines, reproducible experimentation, feature and data management, model registries, testing, serving, monitoring, drift detection, retraining, governance and emerging platform patterns.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Distinguish among MLOps, DataOps, DevOps and model lifecycle management.

2. Design an appropriate CI/CD/CT workflow for machine-learning systems.

3. Explain the role of experiment tracking, reproducibility and model registries.

4. Select suitable tests for data, features, code, models and deployment infrastructure.

5. Compare batch, online, streaming and edge model-serving patterns.

6. Design monitoring strategies for operational performance, data quality, drift, model quality and fairness.

7. Define rational retraining, rollback, promotion and retirement policies.

8. Evaluate organizational, ethical, security and regulatory risks in production ML.

From Experimental Code to a Governed Production System

Traditional software generally maps a defined input to a programmed output. Machine-learning systems include an additional learned component: the model parameters are inferred from data. Consequently, the behaviour of the deployed system depends not only on source code but also on training data, feature definitions, labels, algorithms, hyperparameters, runtime libraries and environmental conditions.

A useful abstraction is:

Production ML system = code + data + configuration + model artefact + infrastructure + operational policies.

A change to any of these elements can alter system behaviour. Updating a feature transformation may cause training-serving skew. A new data source may introduce missing values. A library upgrade may change numerical results. A model trained on a different population may have different fairness characteristics. MLOps therefore treats these elements as versioned and testable dependencies rather than informal parts of an analyst's workspace.

MLOps, DataOps and DevOps

DevOps integrates software development and IT operations through source control, automated testing, continuous integration, deployment automation, observability and shared responsibility. MLOps extends these ideas to systems whose behaviour is learned from data. It adds experiment management, data and feature validation, model evaluation, model registries, drift monitoring and retraining workflows.

DataOps applies automation, testing and collaborative governance to data engineering. Its concerns include data ingestion, schema management, data quality, lineage, pipeline orchestration, access control and reliable delivery of analytical data. DataOps is not merely faster data processing; it is a disciplined approach to making data dependable and discoverable.

The three practices overlap but are not identical. DevOps may verify that an application returns a correct response for a given test input. DataOps verifies that the input data has the expected structure, freshness and semantic quality. MLOps verifies that a model trained or served using that data meets technical and business requirements. A reliable ML system requires all three perspectives.

The Lifecycle as a Feedback System

The model lifecycle is not a linear sequence ending at deployment. It is a feedback loop:

problem definition → data acquisition → preparation and labelling → experimentation → validation → registration → deployment → monitoring → review or retraining → redeployment or retirement.

Feedback enters the loop from several sources: production outcomes, user feedback, data-quality alerts, drift measurements, incidents, new regulations and changes in business objectives. A mature organization makes these feedback paths explicit. It defines who owns them, what evidence is required for action and how changes are recorded.

Experimentation, Reproducibility and Model Lineage

Research notebooks are valuable for exploration, but they are not by themselves a production process. An experiment should be reproducible by recording the dataset version, code commit, environment, feature definitions, random seeds where relevant, hyperparameters, evaluation metrics and resulting artefacts. Reproducibility does not require every stochastic run to produce identical numbers; it requires that another practitioner can understand and repeat the procedure within known tolerances.

Experiment-tracking systems associate runs with parameters, metrics, plots, data references and artefacts. Examples of commonly used technologies include MLflow, Weights & Biases, Kubeflow Metadata and platform-specific tracking services. The technology is less important than the discipline of recording evidence systematically.

Lineage describes the relationships among sources, transformations, features, models, deployments and predictions. For example, a prediction should ideally be traceable to the model version that generated it, the feature values used, the data pipeline run that produced those features and the approval decision that permitted deployment. Such lineage supports debugging, audit, scientific comparison and incident investigation.

Reproducibility also requires environment management. Container images, dependency lock files, infrastructure-as-code and automated build specifications reduce the risk that a model behaves differently between a developer's computer, a validation environment and production. In regulated settings, immutable artefacts and signed packages may be required to establish that the deployed object is the approved object.

Designing CI, CD and CT Pipelines

Continuous Integration

Continuous integration (CI) means that changes are frequently merged into a shared repository and automatically checked. In an ML project, CI should test more than application code. It may include:

• unit tests for feature transformations and utility functions;
• schema and data-contract tests;
• static analysis, dependency and security scans;
• tests for training configuration and pipeline compilation;
• checks that no sensitive data or credentials have entered the repository; and
• small, deterministic training tests to confirm that the pipeline executes.

CI should be fast enough to provide useful feedback. Large-scale training need not occur for every commit. A small representative dataset can test pipeline logic, while more expensive validation is triggered later in the workflow.

Continuous Delivery and Continuous Deployment

Continuous delivery makes a validated release ready for deployment, while continuous deployment automatically releases it when predefined controls pass. In high-risk applications, continuous delivery with human approval may be more appropriate than fully automatic deployment. The choice depends on risk, reversibility, regulatory expectations and the quality of automated evidence.

A typical promotion path is development, validation or staging, canary, and production. A candidate model must satisfy both model criteria and system criteria. A model may have better area under the ROC curve but still be rejected because it exceeds latency limits, uses an unauthorised feature, has unacceptable subgroup performance or cannot be reproduced.

Deployment strategies include:

• Blue-green deployment: two production environments exist; traffic is switched from the old version to the new version after validation.
• Canary deployment: a small proportion of traffic is sent to the candidate before wider release.
• Shadow deployment: the candidate receives production inputs but its outputs are not used for decisions; outputs are compared with the incumbent.
• A/B testing: different versions are exposed to selected user groups, usually when business outcomes can be measured safely.
• Rollback: traffic is returned to a previously approved version when release criteria or operational conditions fail.

Continuous Training

Continuous training (CT) automates some or all of the process of updating a model. A CT pipeline may be initiated by a schedule, a sufficient volume of newly labelled data, a drift alert, a change in business policy or a manual decision. It should not retrain merely because new data exists. The new candidate must be evaluated against an appropriate baseline, tested for data leakage and checked for fairness, stability, resource use and operational compatibility.

A robust CT workflow separates triggering from promotion. An alert or schedule may initiate training, but an independent evaluation and approval policy decides whether the candidate may replace the incumbent. This separation prevents noisy data or malicious input from automatically causing harmful deployment.

DataOps: Making Training and Serving Data Dependable

Data quality is multidimensional. Accuracy, completeness, consistency, timeliness, validity, uniqueness and freshness may all matter, but their importance depends on the use case. A missing value in a non-critical field may be harmless; a missing identity or transaction timestamp may invalidate a prediction.

Data contracts specify expectations between data producers and consumers. They may define schema, data types, allowable ranges, null-handling rules, update frequency, ownership and change-notification procedures. Contracts reduce the likelihood that a silent upstream change will damage a downstream model.

Pipeline orchestration tools such as Airflow, Prefect, Dagster and cloud-native workflow services manage dependencies, retries, scheduling and metadata. Distributed processing systems may be used for large data volumes, while streaming platforms support near-real-time features. The central design principle is that pipelines should be observable and restartable: a failed task should not require manual reconstruction of the entire process.

Feature management introduces another important concern. A feature store or equivalent shared feature layer can provide consistent definitions for training and inference. It commonly includes an offline store for historical training data and an online store for low-latency serving. Point-in-time correctness is essential: a training example must use only information that would have been available at the prediction time. Otherwise, target leakage produces unrealistic evaluation results.

Model Registries and Promotion Policies

A model registry is a controlled catalogue of model versions and their associated metadata. It may record the model artefact, training run, intended use, evaluation results, owner, dependencies, approval status, risk classification, deployment locations and retirement date. Registry stages such as development, candidate, approved, production and archived are useful only when they represent enforceable policies rather than labels applied informally.

Promotion should be evidence-based. A policy might require minimum predictive performance, maximum latency, acceptable calibration error, defined subgroup thresholds, successful security checks and an identified rollback version. Thresholds should be established before reviewing results when possible, because post hoc selection encourages biased decision-making.

The registry also supports separation of duties. The person who trains a model need not be the person who approves it for a regulated decision. Access controls should restrict who can register, approve, deploy, modify or retire models. All such actions should be auditable.

Testing Beyond Accuracy

Testing ML systems requires several complementary layers.

Code and component tests verify ordinary software behaviour, including parsing, transformations, error handling and API contracts. Data tests examine schemas, distributions, missingness, duplicate rates, ranges, freshness and category changes. Feature tests verify transformations, point-in-time joins and consistency between offline and online computation.

Model tests assess predictive performance on holdout or time-based validation data. Depending on the task, metrics may include precision, recall, F1 score, log loss, calibration, ranking measures, mean absolute error or quantile loss. The test design must reflect the deployment context. For temporal prediction, random splitting can leak future information; time-based evaluation is generally more defensible.

Metamorphic tests evaluate whether predictions respond sensibly to controlled transformations. For example, changing an irrelevant field should not change a risk score, while increasing a quantity expected to raise demand should not systematically lower the forecast. Fairness tests compare performance and error patterns across relevant groups. Robustness tests examine missing, noisy, adversarial or out-of-distribution inputs.

Infrastructure and load tests assess throughput, latency, memory use, fault recovery and autoscaling. End-to-end tests confirm that ingestion, feature computation, model invocation, decision logic and logging work together. No single test layer proves that a model is safe; confidence comes from evidence across layers.

Serving Patterns and Operational Architecture

The serving pattern should follow decision latency, data availability, cost and reliability requirements.

In batch serving, predictions are generated periodically and stored for later use. This is suitable for daily marketing lists, scheduled risk reports or inventory planning. It is relatively simple and economical but cannot respond immediately to new events.

In online serving, an API receives a request and returns a prediction within a defined service-level objective. It requires request validation, feature retrieval, model loading, timeouts, authentication, rate limiting and observability. A model server may expose REST or gRPC interfaces, while a feature service supplies low-latency inputs.

Streaming inference processes events continuously, often using message brokers and stream processors. It is useful for fraud detection, equipment monitoring and real-time recommendations. The architecture must address ordering, late events, duplicate messages and state recovery.

Edge inference executes the model near the data source, such as on a mobile device or industrial controller. It can reduce latency and protect sensitive data, but model size, hardware variation, update distribution and security become significant concerns.

A production service should define service-level indicators such as availability, latency percentiles, error rate, throughput and resource consumption. A model with excellent statistical performance is not operationally successful if it times out during peak demand.

Monitoring: Observing the Model and Its Environment

Monitoring should be designed before deployment. It is useful to distinguish five layers.

1. System monitoring measures availability, CPU and memory usage, latency, throughput, queue depth and failures.

2. Data-quality monitoring measures schema changes, null rates, invalid values, freshness, duplicates and volume anomalies.

3. Drift monitoring compares production inputs or predictions with a reference distribution.

4. Model-performance monitoring compares predictions with later-arriving labels and tracks relevant metrics.

5. Business and responsible-use monitoring examines outcomes such as approval rates, losses, customer complaints, safety events and subgroup disparities.

Drift is not a single phenomenon. Data drift is a change in the distribution of input variables. Concept drift is a change in the relationship between inputs and the target. Prediction drift is a change in the distribution of outputs. Label drift is a change in the target prevalence. A model can experience input drift without performance deterioration, or performance deterioration without an easily detectable change in marginal inputs.

Statistical measures such as the population stability index, Wasserstein distance, Kullback–Leibler divergence and distributional tests can support detection. However, thresholds require contextual interpretation. A statistically significant change in a high-volume dataset may have no practical importance, while a small change in a safety-critical variable may matter greatly. Alerts should therefore combine statistical evidence, business impact and operational judgement.

Delayed labels create a special challenge. If loan default is observed months after approval, immediate model-quality monitoring is impossible. Organizations may use proxy indicators, monitor input and prediction behaviour in the short term, and evaluate definitive performance when labels mature. Monitoring must also account for feedback loops: predictions can influence decisions, and those decisions can alter the future data used for training.

Retraining, Rollback and Retirement

Retraining policies should specify the trigger, data window, validation method, approval authority and rollback plan. Common triggers include a sustained performance decline, a drift threshold, a change in product or policy, accumulation of new labels, or a fixed review interval. Trigger-based retraining is preferable to blindly retraining at arbitrary intervals, but it must be protected against alert noise and transient events.

The incumbent model should remain available until the candidate has passed all gates. Champion–challenger evaluation compares the production champion with one or more candidate models under the same conditions. If a candidate fails after deployment, rollback should be rapid and tested rather than theoretical. Model, feature and code versions must be compatible with the rollback artefact; reverting only the model may not restore the previous behaviour if upstream features have changed.

Retirement is an active lifecycle decision. A model may be retired because its use case has ended, its data source is no longer permitted, its performance is inadequate, its replacement is superior, or its risks cannot be controlled. Retirement requires preserving appropriate records, communicating dependencies, disabling endpoints and ensuring that downstream systems do not silently continue using the old version.

Governance, Security and Responsible Operation

Model governance connects technical controls with organizational accountability. Documentation should explain intended use, excluded use, training population, known limitations, evaluation design, monitoring plan and responsible owners. A model card or comparable document can communicate this information, while a data sheet can describe data sources, collection conditions and limitations.

Privacy requirements may affect data retention, feature selection, access rights and logging. Production logs should not expose unnecessary personal or sensitive information. Encryption, secret management, network controls, least-privilege access, dependency scanning and image signing are practical security measures. ML-specific threats include data poisoning, evasion attacks, model extraction, prompt or input manipulation and leakage through prediction APIs.

Fairness cannot be guaranteed by a single metric. Different definitions may conflict, and an apparently neutral feature may encode historical inequality. Teams should identify affected groups, evaluate error and outcome disparities, involve domain experts and document trade-offs. Human review is not automatically a safeguard if reviewers lack authority, time or adequate information.

Regulatory and professional obligations vary by sector and jurisdiction. Nevertheless, sound practice generally requires lawful data use, purpose limitation, transparency appropriate to the decision, contestability where relevant, auditability and meaningful human accountability. Automation should not obscure responsibility: an organization remains accountable for decisions made through a model it deploys.

Illustrative Case Study: A Retail Fraud-Detection Platform

Background

A fictional national online retailer uses a machine-learning model to assign a fraud risk score to card-not-present transactions. High-risk transactions are blocked or routed to manual review. The original model was trained on historical transactions and deployed as an online API.

Situation and Problem

During a major promotional period, the proportion of transactions sent to manual review rises sharply. Customer complaints increase, while the fraud-loss rate initially appears stable. Engineers discover that a payment provider has changed the format and timing of several fields. At the same time, fraudsters have adopted a new purchasing pattern that was uncommon in the training data.

Analysis

The organization first separates system, data and model questions. API latency and availability are normal, so the incident is not primarily an infrastructure failure. Data-quality checks reveal a sudden increase in missing values and a change in category encoding. Input monitoring also identifies drift in device and transaction-velocity features. Because confirmed fraud labels arrive after investigation, definitive model-performance metrics are delayed. However, the manual-review rate and customer-abandonment rate provide immediate business signals.

A lineage review identifies that the online feature transformation was maintained separately from the historical training transformation. This created training-serving skew. The model registry shows that the deployed version was approved before the payment provider's change, but the release record does not include an explicit dependency on the provider schema.

Approach and Solution

The retailer temporarily lowers the exposure of the affected model through a canary-style traffic policy and restores a previous model-compatible feature path where possible. It adds a schema contract with the payment provider and blocks deployment when required fields or category semantics change. The team consolidates feature definitions into a governed feature layer with point-in-time historical reconstruction.

A new candidate is trained using recent, reviewed cases and evaluated with time-based validation. Evaluation includes fraud recall, false-positive rate, calibration, latency, subgroup checks and performance under simulated missingness. Shadow deployment compares candidate and incumbent outputs without immediately changing customer decisions. The candidate is promoted only after business and risk owners approve the evidence.

The monitoring design is revised to include data-quality alerts, prediction-rate limits, review-volume thresholds, delayed-label performance tracking and a documented incident response procedure. Retraining is triggered by sustained deterioration after labels mature, not by every short-lived distribution fluctuation.

Outcome

In this illustrative case, the retailer restores stable review volumes, improves traceability and reduces the probability that an upstream schema change will silently alter decisions. The principal improvement is not simply a new algorithm; it is a controlled lifecycle connecting data contracts, feature consistency, deployment gates, monitoring and accountable response.

Learning Points

The case demonstrates that model failure may originate outside the model code. It also shows why business metrics, delayed labels, lineage and rollback capability must be considered together. A drift alert is useful only when the organization has a defined interpretation and response. Finally, continuous training without data governance and approval controls could have amplified the incident rather than solving it.

Challenges and Common Failure Modes

A frequent mistake is to treat MLOps as a tool-purchasing exercise. A registry or orchestration platform cannot compensate for unclear ownership, weak evaluation design or undocumented business requirements. Another mistake is to optimize only for predictive accuracy while ignoring calibration, fairness, latency, cost and maintainability.

Teams may also create excessive automation. Automatically retraining and deploying after every alert can produce unstable model behaviour, especially when labels are delayed or data is temporarily corrupted. Conversely, excessive manual approval can make the lifecycle too slow and encourage bypassing formal controls.

Other common failures include training-serving skew, leakage from future information, unversioned data, untested rollback procedures, monitoring only infrastructure metrics, alert fatigue, inadequate access control and retaining sensitive logs indefinitely. Fragmented ownership between data engineering, data science, software engineering, operations and business teams creates gaps at system boundaries.

MLOps also involves economic trade-offs. A highly available online service, real-time feature store and extensive monitoring may be justified for fraud prevention but excessive for a monthly forecasting report. Architecture should be proportionate to risk, volume, latency and the cost of failure.

Emerging Directions and Best Practices

Modern platforms increasingly provide managed experiment tracking, feature stores, model registries, container orchestration, policy enforcement and observability in integrated environments. Git-based workflows and infrastructure-as-code improve reviewability and reproducibility. Data and model quality tools are becoming more closely connected, enabling a single lineage view from source data to prediction.

Generative AI introduces additional lifecycle concerns, including prompt and retrieval versioning, evaluation of factuality and harmful outputs, context-data governance, token and latency monitoring, and protection against prompt injection. The same principles remain relevant: version dependencies, test realistic failure modes, monitor actual use and maintain accountable release controls.

Effective practice can be summarized as follows:

• define the business decision, risk tolerance and success measures before training;
• version code, data references, features, configurations and environments;
• automate repeatable tests, but retain risk-based human judgement;
• separate training, evaluation, approval and deployment responsibilities where appropriate;
• use time-aware and leakage-resistant validation;
• monitor technical, statistical, business and responsible-use indicators;
• make rollback, incident response and retirement operationally real;
• document assumptions, limitations, ownership and intended use; and
• design the platform according to the consequences of failure rather than fashion.

Chapter Summary

MLOps, DataOps and model lifecycle management transform machine learning from an isolated experiment into a dependable organizational capability. MLOps extends DevOps with data, model and experiment concerns, while DataOps establishes reliable, governed and observable data delivery. The lifecycle includes data preparation, experimentation, validation, registration, deployment, monitoring, retraining and retirement.

CI/CD/CT pipelines automate integration, delivery and controlled model updating. Reproducibility depends on versioned data references, code, environments, features and artefacts. Registries provide controlled promotion and lineage. Testing must cover code, data, features, models, infrastructure, fairness and end-to-end behaviour. Serving may be batch, online, streaming or edge-based, according to operational requirements.

Production monitoring combines system health, data quality, drift, delayed model performance, business outcomes and responsible-use indicators. Retraining should be triggered by evidence and separated from automatic promotion. Governance, privacy, security, auditability and human accountability are essential, particularly in high-impact applications. Reliable ML is therefore not only an engineering problem; it is a socio-technical practice involving architecture, process, risk management and organizational responsibility.

Key Terms

MLOps: Practices for developing, deploying, monitoring and governing machine-learning systems in production.

DataOps: Collaborative and automated practices for reliable, testable and governed data delivery.

DevOps: Integration of software development and IT operations through automation and shared responsibility.

Continuous Integration (CI): Frequent integration of code changes accompanied by automated validation.

Continuous Delivery (CD): Automated preparation of validated software or model releases for deployment.

Continuous Training (CT): Automated or semi-automated retraining and evaluation of models as data or conditions change.

Model registry: A controlled catalogue of model versions, artefacts, metadata, approvals and deployment status.

Model lineage: Traceability of relationships among data, transformations, experiments, models and predictions.

Data contract: A formal agreement specifying the schema, quality, semantics and delivery expectations of data.

Feature store: A managed system for defining, storing and serving reusable machine-learning features.

Training-serving skew: A difference between the feature computation used during training and that used during inference.

Data drift: Change in the distribution of model input data over time.

Concept drift: Change in the relationship between input variables and the target outcome.

Prediction drift: Change in the distribution of model outputs.

Champion model: The currently approved production model used as the reference for comparison.

Challenger model: A candidate model evaluated against the production champion.

Canary deployment: Gradual release of a candidate to a small proportion of production traffic.

Shadow deployment: Running a candidate on production inputs without using its outputs for decisions.

Point-in-time correctness: Use of only information available at the prediction time when constructing training features.

Model rollback: Reversion to a previously approved model and compatible system configuration.

Review Questions

Short Answer Questions

1. Why does a machine-learning system require more versioning than ordinary application code?

2. Distinguish between continuous delivery and continuous training.

3. What is training-serving skew, and why is it dangerous?

4. Define data drift and concept drift.

5. State two responsibilities of a model registry.

Long Answer Questions

1. Explain how CI/CD/CT pipelines can be designed for a high-risk machine-learning application.

2. Discuss the role of experiment tracking, reproducibility and lineage in professional ML practice.

3. Compare batch, online, streaming and edge serving architectures, including their trade-offs.

4. Explain the layers of production monitoring and how delayed labels affect monitoring design.

5. Describe how governance, security, privacy and fairness considerations should be integrated into the model lifecycle.

Analytical / Application Questions

1. A demand-forecasting model shows substantial input drift, but current business error remains stable. What evidence and actions would you require before initiating retraining?

2. An online model has improved predictive performance but increased latency and subgroup false-positive disparity. Should it be promoted? Develop a decision framework.

3. Design a rollback and incident-response procedure for a fraud model whose upstream data provider has unexpectedly changed its schema.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes the relationship between MLOps and DevOps?

A. MLOps replaces DevOps because machine-learning systems do not require software engineering.

B. MLOps extends DevOps practices with data, experiment, model, drift and retraining concerns.

C. MLOps is limited to deploying notebooks as APIs.

D. MLOps concerns only the purchase of cloud infrastructure.

Correct Answer: B. MLOps extends DevOps practices with data, experiment, model, drift and retraining concerns.

Question 2

A model is trained using a feature computed from transactions occurring after the prediction timestamp. Which problem has most likely occurred?

A. Canary failure

B. Prediction drift

C. Target or temporal leakage

D. Infrastructure autoscaling

Correct Answer: C. Target or temporal leakage

Question 3

What is the primary purpose of separating a continuous-training trigger from model promotion?

A. To prevent any future model from being evaluated.

B. To ensure that every retrained model is deployed immediately.

C. To allow new candidates to be evaluated and approved before replacing the incumbent.

D. To eliminate the need for monitoring.

Correct Answer: C. To allow new candidates to be evaluated and approved before replacing the incumbent.

Question 4

Which deployment method runs a candidate model on production inputs while its outputs are withheld from operational decisions?

A. Blue-green deployment

B. Shadow deployment

C. Batch deployment

D. Rollback deployment

Correct Answer: B. Shadow deployment

Question 5

A payment provider changes the meaning of a categorical field, but the pipeline continues without an error. Which control would most directly address this risk?

A. A data contract with schema and semantic validation

B. Increasing the model's number of layers

C. Reducing API timeout values

D. Deleting experiment metadata

Correct Answer: A. A data contract with schema and semantic validation

Question 6

Which observation is the strongest example of concept drift?

A. The percentage of null values in an input feature increases.

B. The distribution of model scores changes while labels remain unavailable.

C. The relationship between customer behaviour variables and fraud outcomes changes over time.

D. The model-serving container uses more memory.

Correct Answer: C. The relationship between customer behaviour variables and fraud outcomes changes over time.

Question 7

Why is point-in-time correctness important in feature construction?

A. It guarantees that the model will have perfect accuracy.

B. It prevents training features from using information unavailable when the historical prediction would have been made.

C. It removes the need for an online feature store.

D. It ensures that all features have identical numerical scales.

Correct Answer: B. It prevents training features from using information unavailable when the historical prediction would have been made.

Question 8

Which item is an operational rather than purely predictive acceptance criterion for a candidate model?

A. Recall on a holdout set

B. Calibration error

C. Ninety-fifth-percentile inference latency

D. Subgroup false-negative rate

Correct Answer: C. Ninety-fifth-percentile inference latency

Question 9

A team automatically deploys a newly retrained model whenever a drift alert occurs. What is the principal weakness of this approach?

A. Drift can never be measured statistically.

B. Drift may reflect temporary or corrupted data and does not by itself prove that a new model is safe to deploy.

C. Retraining cannot use recent labels.

D. Production models should never be monitored.

Correct Answer: B. Drift may reflect temporary or corrupted data and does not by itself prove that a new model is safe to deploy.

Question 10

Which practice most effectively supports recovery when a new model and feature transformation cause production failures?

A. Keeping only the latest model artefact

B. Removing deployment records after release

C. Maintaining versioned, compatible model, feature, code and environment artefacts with a tested rollback procedure

D. Relying on the developer's local notebook

Correct Answer: C. Maintaining versioned, compatible model, feature, code and environment artefacts with a tested rollback procedure