13. Data, AI, MLOps and Intelligent Operations

Introduction and Context

Cloud platforms have evolved from infrastructure providers into integrated environments for data engineering, artificial intelligence, machine learning, observability, and operational automation. DevOps teams increasingly manage systems whose behaviour depends not only on application code and infrastructure configuration, but also on data quality, statistical models, feature pipelines, and automated decisions. Consequently, modern cloud engineering requires a unified understanding of data platforms, machine learning operations (MLOps), and intelligent operations (AIOps).

A conventional DevOps pipeline moves source code through build, test, release, and deployment stages. An MLOps pipeline must additionally manage training data, labels, features, experiments, model versions, evaluation metrics, deployment environments, and model drift. AIOps applies analytics and machine learning to operational telemetry in order to detect anomalies, correlate events, predict failures, and assist incident response. Generative AI extends these capabilities by summarising incidents, explaining system behaviour, generating queries and automation scripts, and supporting engineers through natural-language interfaces.

The central challenge is balance. Automation can improve speed and reliability, but poorly governed data or models can amplify errors, create privacy violations, and obscure accountability. This chapter examines the architecture, lifecycle, applications, governance, and strategic implications of cloud data, AI, MLOps, and intelligent operations across AWS, Google Cloud, and Microsoft Azure.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the principal cloud services and architectural patterns used for data engineering and analytics.

2. Distinguish data warehouses, data lakes, lakehouses, streaming platforms, and feature stores.

3. Describe the stages of an MLOps lifecycle and the role of automation in each stage.

4. Design a high-level data and ML pipeline for a cloud-based DevOps environment.

5. Apply AIOps techniques to anomaly detection, event correlation, incident analysis, and remediation.

6. Evaluate the use of generative AI in operations, including its benefits and limitations.

7. Identify risks involving bias, privacy, security, explainability, model drift, and regulatory compliance.

8. Select appropriate governance and operational practices for reliable intelligent systems.

From Cloud Data to Operational Intelligence

Data architecture begins with sources. These may include application databases, APIs, logs, traces, IoT devices, SaaS systems, and infrastructure metrics. Data is commonly classified as structured, semi-structured, or unstructured. Structured data fits predefined schemas; semi-structured data includes formats such as JSON; unstructured data includes documents, images, audio, and free-form text. The classification influences storage, processing, governance, and analytical methods.

Cloud platforms separate storage from computation. Object storage—Amazon S3, Google Cloud Storage, and Azure Blob Storage—provides durable, scalable storage for raw and processed files. Managed relational and analytical services include Amazon Redshift, Google BigQuery, and Azure Synapse Analytics. Managed processing services include AWS Glue and EMR, Google Dataflow and Dataproc, and Azure Data Factory, Synapse pipelines, and Databricks integrations. Streaming services include Amazon Kinesis, Google Pub/Sub, and Azure Event Hubs.

A data pipeline normally performs ingestion, validation, transformation, storage, serving, and monitoring. Batch pipelines process accumulated data at scheduled intervals, whereas streaming pipelines process events continuously or in small time windows. Batch processing is often simpler and more economical for periodic reporting. Streaming is valuable when decisions depend on low latency, such as fraud detection, service monitoring, or real-time personalisation.

Data quality is not merely a database concern. Completeness, accuracy, consistency, timeliness, uniqueness, and validity affect dashboards, models, and operational decisions. Data contracts make expectations explicit between producers and consumers by defining schemas, ownership, acceptable values, and change procedures. Schema registries and automated validation reduce the risk that an upstream change silently breaks downstream systems.

Warehouses, Lakes, and Lakehouses

A data warehouse stores curated, structured data for reporting and analytical queries. It normally applies schema-on-write: data is validated and organised before it is stored for consumption. Warehouses provide strong performance, governance, and support for business intelligence, but can be less flexible for raw files, large-scale experimentation, and machine learning workloads.

A data lake stores raw data in inexpensive, scalable storage and supports many formats. It applies schema-on-read, allowing different consumers to interpret data for different purposes. Without cataloguing, access controls, quality rules, and lifecycle policies, however, a lake can become a “data swamp”—a collection of poorly understood and unreliable assets.

A lakehouse combines the flexibility of a lake with warehouse-like reliability. Transactional table formats, metadata layers, versioning, schema enforcement, and governance permit analytical and ML workloads to use common data. The lakehouse is not simply a product; it is an architectural pattern that seeks to reduce duplication and improve consistency across data engineering, analytics, and data science.

A typical lakehouse design separates zones. The bronze or raw zone preserves source data with minimal alteration. The silver or refined zone applies cleansing, standardisation, and deduplication. The gold or serving zone contains business-ready datasets, features, aggregates, or data products. These zones should be governed by ownership, retention policies, lineage, and access controls rather than treated as an automatic quality guarantee.

Data Pipelines and Analytical Workflows

A reliable pipeline is an engineered product rather than a collection of scripts. Orchestration tools schedule dependencies, retry transient failures, record execution state, and expose operational metrics. A useful pipeline has idempotent operations: rerunning a failed task should not create duplicate or inconsistent results. It also separates configuration from code and stores secrets in managed secret-management systems.

A mature workflow includes the following controls:

• source validation and schema compatibility checks;
• data quality tests and quarantine of invalid records;
• lineage from source to report or model feature;
• encryption in transit and at rest;
• role-based or attribute-based access control;
• freshness, volume, distribution, and completeness monitoring;
• reproducible transformations and versioned definitions; and
• controlled retention, deletion, and archival.

Data observability extends application observability to datasets and pipelines. It asks whether data arrived on time, whether its volume changed unexpectedly, whether values remain within plausible ranges, and whether relationships between fields have changed. These signals are important because a model can remain technically available while producing poor decisions from degraded input data.

The MLOps Lifecycle

MLOps applies DevOps principles to machine learning while recognising that models depend on data and statistical behaviour. The lifecycle is iterative rather than strictly linear.

Problem Framing and Data Preparation

The process begins with a business or operational objective, such as predicting service demand or identifying anomalous transactions. The objective must be translated into a measurable target and an acceptable cost of errors. Data is collected, labelled where necessary, cleaned, explored, and divided into training, validation, and test sets. Time-dependent problems require temporal splits to avoid training on information that would not have been available at prediction time.

Data leakage is a serious error. It occurs when information related to the target enters training in a way that would not exist during real prediction. Leakage can produce impressive test scores and poor production performance. Data preparation should therefore be implemented as a reproducible pipeline, not as undocumented notebook actions.

Experimentation and Training

Training involves selecting algorithms, features, hyperparameters, and evaluation methods. An experiment-tracking system records code versions, datasets, parameters, metrics, environment details, and artefacts. A model registry manages model versions and their promotion through stages such as development, validation, staging, and production.

Evaluation must reflect operational consequences. Accuracy alone is inadequate for imbalanced classification; precision, recall, F1 score, area under relevant curves, calibration, latency, cost, and group-level performance may all matter. A model that is statistically strong but too slow or expensive for its service-level objective is not operationally successful.

Deployment and Serving

Models may be deployed as online endpoints, batch scoring jobs, embedded components, or edge services. Deployment strategies include shadow deployment, canary release, blue-green release, and gradual traffic shifting. Shadow deployment compares predictions without influencing users. Canary deployment exposes a new model to a small proportion of traffic and evaluates technical and business indicators before expansion.

The serving layer must define input and output schemas, timeouts, authentication, rate limits, fallback behaviour, and logging. It should avoid logging sensitive payloads unnecessarily. A model endpoint is a production service and must be managed with the same discipline as any other critical API.

Monitoring, Feedback, and Retirement

MLOps monitoring includes infrastructure health, service latency, error rates, prediction distributions, data quality, concept drift, and outcome performance. Data drift describes changes in input distributions; concept drift describes changes in the relationship between inputs and outcomes. A drift alert is a signal for investigation, not an automatic justification for retraining.

Retraining may be scheduled, triggered by drift, or initiated by new labelled outcomes. Automated retraining requires safeguards against poisoning, feedback loops, accidental degradation, and unreviewed promotion. Models and data should have retirement criteria, ownership, documentation, and an audit trail.

Cloud Services and Reference Architecture

AWS commonly combines S3, Glue, Lake Formation, Athena, Redshift, Kinesis, SageMaker, CloudWatch, and IAM. Google Cloud commonly combines Cloud Storage, BigQuery, Dataflow, Pub/Sub, Dataplex, Vertex AI, Cloud Monitoring, and IAM. Azure commonly combines Blob Storage or ADLS, Data Factory, Synapse, Event Hubs, Azure Machine Learning, Monitor, Purview, and Entra ID. Equivalent services differ in terminology and integration details, but the architectural responsibilities are comparable.

A reference architecture may contain event producers, a streaming or batch ingestion layer, object storage and catalogue services, transformation and quality jobs, an analytical serving layer, a feature-management layer, a training platform, a model registry, an inference service, and observability systems. Security services provide identity, key management, network controls, policy enforcement, and audit logs. Infrastructure as code defines the repeatable deployment of these components.

The correct choice is not determined by brand alone. Architects should evaluate data volume and velocity, latency requirements, portability, skills, integration, governance, cost, vendor lock-in, and operational maturity. Managed services reduce undifferentiated maintenance but may create platform-specific dependencies.

AIOps and Intelligent Operations

AIOps applies statistical analysis, machine learning, automation, and knowledge management to IT operations. Its main data sources are logs, metrics, traces, deployment events, tickets, topology information, configuration records, and user-impact indicators.

Anomaly detection establishes a representation of normal behaviour and identifies observations that significantly differ from it. Methods range from static thresholds and seasonal baselines to clustering, forecasting, isolation techniques, and neural models. Static thresholds remain useful when domain limits are clear. Machine learning is more valuable where behaviour is seasonal, multivariate, or difficult to specify manually.

Event correlation reduces alert noise by grouping signals that share time, topology, service, or causal relationships. A database failure may generate alerts from many applications; correlation should help identify the probable initiating event rather than treating every symptom as a separate incident. Correlation is not proof of causation, so engineers need supporting evidence and confidence indicators.

Incident analysis combines telemetry, change history, service dependencies, runbooks, and historical incidents. Generative AI can summarise an incident timeline, identify relevant logs, propose hypotheses, draft a post-incident review, and translate natural-language questions into queries. Retrieval-augmented generation (RAG) grounds responses in approved runbooks, architecture documents, and recent telemetry rather than relying only on a model’s general training.

Remediation may be manual, approval-based, or fully automated. Safe automation is bounded by conditions such as service ownership, blast-radius limits, rate limits, rollback procedures, and approval requirements. Examples include restarting a failed worker, scaling a service within limits, rotating a compromised credential, or disabling a faulty feature flag. Destructive or irreversible actions should generally require human confirmation.

Generative AI in DevOps and MLOps

Generative AI supports code explanation, test generation, infrastructure-template drafting, query construction, documentation, incident communication, and knowledge retrieval. Its value is greatest when it reduces cognitive load while leaving engineers responsible for verification.

A production design should define an approved model, data-handling rules, prompt templates, grounding sources, output filters, evaluation datasets, and audit mechanisms. Sensitive logs should be minimised or masked before being sent to a model. Outputs must be treated as suggestions because generative models can hallucinate, misunderstand context, reveal confidential content, or generate insecure code.

Useful evaluation dimensions include factual accuracy, groundedness, completeness, refusal behaviour, latency, cost, security, and usefulness to engineers. Prompt injection is particularly relevant when a model reads untrusted logs or tickets containing instructions designed to manipulate it. Tool access should therefore be restricted, parameterised, and audited.

Governance, Risk, and Responsible Use

AI governance links technical controls with organisational accountability. A model should have an owner, purpose statement, risk classification, approved data sources, performance thresholds, monitoring plan, and retirement process. Model cards, data documentation, decision logs, and lineage support transparency.

Bias may arise from unrepresentative samples, historical discrimination, proxy variables, measurement errors, or unequal label quality. Fairness analysis should examine relevant groups and error types, but no single fairness metric is universally correct. The organisation must determine which harms matter and how trade-offs will be managed.

Privacy requires purpose limitation, data minimisation, lawful processing, retention controls, access restriction, and appropriate de-identification. Anonymisation is difficult when multiple datasets can be linked. Encryption does not remove the need for governance. Sensitive data may also appear in logs, prompts, model outputs, embeddings, backups, and feature stores.

Model risk includes inaccurate predictions, drift, adversarial manipulation, data poisoning, insecure dependencies, overreliance, and failure to explain decisions. Human oversight should be proportionate to impact. High-impact decisions require stronger validation, review, appeal, documentation, and segregation of duties than low-risk operational recommendations.

Illustrative Case: Intelligent Operations for a Digital Commerce Platform

Background

An illustrative online commerce company operates services for checkout, inventory, payments, and delivery estimation across several cloud regions. It receives millions of application events daily and uses both batch analytics and real-time monitoring.

Situation and Problem

The operations team receives approximately 20,000 alerts per week. Many are duplicate symptoms from the same underlying failure. During a previous incident, a deployment increased checkout latency, but the problem was identified only after customer complaints. The company proposes an AIOps system that correlates telemetry, predicts anomalies, and recommends remediation.

Analysis

Architects identify four requirements: low-latency detection, reliable historical analysis, protection of payment data, and human accountability. Raw logs and events are streamed to durable object storage and a streaming platform. A lakehouse stores raw, refined, and serving datasets. Metrics and traces remain available through the observability platform, while selected operational data is replicated for correlation. Data contracts are introduced for service events.

The team discovers that alert volume is partly caused by inconsistent service names and missing deployment metadata. It therefore improves instrumentation before training an anomaly model. A baseline forecasting model is first used for latency and error-rate anomalies. More complex models are postponed until data quality and evaluation procedures are stable.

Approach and Solution

A model registry records experiments and approved versions. Shadow deployment evaluates anomaly scores without changing alert routing. Event correlation uses service topology, timestamps, deployment records, and known dependencies. A retrieval-based assistant uses approved runbooks and incident histories to produce an evidence-linked incident summary.

Automated remediation is limited to reversible actions: scaling within a defined range and restarting unhealthy workers. Credential changes, database failover, and customer-impacting feature changes require approval. Every recommendation records the evidence used, the user who approved it, the action taken, and the result.

Outcome

The organisation reports fewer duplicate alerts, faster initial triage, and more consistent incident documentation in its internal evaluation. It also identifies false positives during seasonal sales events and adds event calendars and workload forecasts to the monitoring context. The system is treated as an operational aid rather than an autonomous authority.

Learning Points

The case demonstrates that AIOps success depends on instrumentation, data quality, service topology, and governance. Machine learning cannot compensate for ambiguous ownership or incomplete telemetry. Gradual adoption, bounded automation, and measurable evaluation are safer than immediate full autonomy.

Challenges and Common Mistakes

Common failures include building models before defining the operational decision, confusing correlation with causation, training on leaked data, ignoring class imbalance, deploying without rollback, and monitoring infrastructure while neglecting data and model behaviour. Other mistakes include unrestricted access to sensitive logs, unreviewed generative-AI outputs, unversioned notebooks, excessive alerting, and retraining whenever drift is detected without investigating its cause.

Cost is also a design concern. Continuous streaming, high-cardinality telemetry, repeated training, and large-language-model inference can become expensive. Sampling, tiered retention, efficient feature computation, batching, caching, and clear service-level objectives help manage expenditure. Portability should be considered where proprietary formats, APIs, or model services create switching costs.

Current Developments and Best Practices

Contemporary practice is moving toward unified observability, data products, real-time lakehouse processing, confidential computing, smaller specialised language models, and policy-driven AI platforms. Digital twins and causal techniques are being explored for simulation and operational planning, although their value depends on accurate system representations.

Effective practice can be summarised as follows:

• begin with a measurable decision and defined business or operational loss;
• establish ownership, data contracts, lineage, and quality checks;
• treat pipelines, features, models, prompts, and infrastructure as versioned artefacts;
• automate testing, security scanning, deployment, monitoring, and rollback;
• use progressive delivery and human approval for high-risk actions;
• evaluate performance across time, environments, and relevant user groups;
• minimise sensitive data and record access and decision evidence;
• monitor cost, latency, reliability, drift, and user impact; and
• review incidents and retire models or automations that no longer provide value.

Chapter Summary

Cloud data platforms provide the foundation for analytics, machine learning, and intelligent operations. Warehouses support governed analytical workloads, lakes provide flexible raw storage, and lakehouses combine scalable storage with transactional and governance capabilities. Reliable pipelines require validation, lineage, quality monitoring, security, and reproducibility.

MLOps extends DevOps by managing data, experiments, features, models, deployment, monitoring, retraining, and retirement. AIOps applies analytics to telemetry and operational knowledge for anomaly detection, event correlation, incident analysis, and bounded remediation. Generative AI can assist with investigation and automation when responses are grounded, evaluated, secured, and reviewed.

The technical benefits of intelligent operations are inseparable from governance. Bias, privacy, drift, security, explainability, cost, and accountability must be addressed throughout the lifecycle. The most dependable approach combines strong engineering foundations, progressive automation, measurable outcomes, and proportionate human oversight.

Key Terms

Data lake: Scalable storage for raw and varied data, usually using schema-on-read.

Data warehouse: Governed analytical store containing structured, curated data.

Lakehouse: Architecture combining data-lake flexibility with warehouse-like reliability and governance.

Data contract: Agreement defining a dataset’s schema, quality, ownership, and change expectations.

Data lineage: Record of how data moves and changes from source to consumption.

Data observability: Monitoring of data freshness, volume, quality, distribution, and pipeline health.

Feature: Input variable prepared for use by a machine-learning model.

Feature store: System for managing, serving, and governing reusable ML features.

MLOps: Practices for developing, deploying, monitoring, and governing machine-learning systems.

Model drift: Change that reduces model validity as data or relationships evolve.

Data leakage: Use of information in training that would not be available at prediction time.

Model registry: Repository tracking model versions, metadata, approvals, and deployment stages.

AIOps: Application of analytics and AI to IT operations and service management.

Anomaly detection: Identification of observations that differ significantly from expected behaviour.

Event correlation: Grouping related operational signals to reduce noise and identify probable causes.

RAG: Retrieval-augmented generation that grounds model output in retrieved approved information.

Concept drift: Change in the relationship between input variables and the target outcome.

Model risk: Potential harm caused by inaccurate, insecure, biased, or improperly governed models.

Data poisoning: Deliberate or accidental contamination of training data to influence a model.

Progressive delivery: Gradual release of software or models using controlled exposure and monitoring.

Review Questions

Short Answer Questions

1. What is the principal architectural difference between a data warehouse and a data lake?

2. Why are data contracts useful in cloud data pipelines?

3. Distinguish data drift from concept drift.

4. What is the purpose of a model registry?

5. Why should generative-AI outputs in operations be verified by engineers?

Long Answer Questions

1. Explain the lakehouse pattern and discuss how bronze, silver, and gold zones support governance and analytics.

2. Describe the complete MLOps lifecycle from problem framing to model retirement.

3. Explain how AIOps performs anomaly detection, event correlation, and incident analysis.

4. Compare batch and streaming data pipelines, including their operational trade-offs.

5. Discuss privacy, bias, explainability, and accountability in cloud-based AI systems.

Analytical / Application Questions

1. Design a high-level MLOps architecture for predicting application capacity demand across multiple cloud regions. Identify data sources, pipeline controls, model monitoring, and rollback procedures.

2. An organisation receives thousands of duplicate alerts after every deployment. Analyse likely causes and propose an AIOps approach that combines telemetry, topology, change data, and human approval.

3. A generative-AI assistant recommends deleting a storage resource after reading an incident log. Explain the risks and design controls that would prevent unsafe execution while preserving useful automation.

22. Multiple Choice Questions with Answers

Question 1

Which characteristic most clearly distinguishes a lakehouse from a traditional data lake?

A. It stores only structured data.

B. It combines flexible large-scale storage with transactional reliability and governance.

C. It eliminates the need for data quality controls.

D. It processes only real-time events.

Correct Answer: B. It combines flexible large-scale storage with transactional reliability and governance.

Question 2

A model performs exceptionally well in testing but poorly in production because future information was included in its training features. What is this problem called?

A. Concept drift

B. Event correlation

C. Data leakage

D. Under-sampling

Correct Answer: C. Data leakage

Question 3

Which practice best supports an idempotent data pipeline?

A. Appending output blindly whenever a task is retried.

B. Ensuring that rerunning a task produces the same intended result without duplicates.

C. Removing all validation to improve throughput.

D. Storing credentials in transformation scripts.

Correct Answer: B. Ensuring that rerunning a task produces the same intended result without duplicates.

Question 4

What is the main purpose of a model registry in MLOps?

A. To replace all production monitoring.

B. To store only raw application logs.

C. To track model versions, metadata, approvals, and deployment stages.

D. To automatically guarantee fairness.

Correct Answer: C. To track model versions, metadata, approvals, and deployment stages.

Question 5

An increase in the distribution of incoming transaction amounts, without yet knowing whether prediction accuracy has changed, is an example of what?

A. Data drift

B. Concept drift

C. Data poisoning

D. Schema enforcement

Correct Answer: A. Data drift

Question 6

Which AIOps technique is most directly intended to reduce multiple symptom alerts generated by one underlying failure?

A. Event correlation

B. Data anonymisation

C. Model retirement

D. Batch scoring

Correct Answer: A. Event correlation

Question 7

Why is retrieval-augmented generation useful in an incident assistant?

A. It guarantees that the language model cannot make errors.

B. It grounds responses in approved runbooks, architecture information, or incident records.

C. It removes the need for access controls.

D. It prevents all prompt injection attacks automatically.

Correct Answer: B. It grounds responses in approved runbooks, architecture information, or incident records.

Question 8

Which remediation policy is most appropriate for a high-impact, potentially irreversible database failover?

A. Execute it automatically whenever any anomaly appears.

B. Permit it only after defined evidence and human approval.

C. Disable all monitoring before execution.

D. Allow any user to trigger it without an audit record.

Correct Answer: B. Permit it only after defined evidence and human approval.

Question 9

Which evaluation approach is most appropriate for an imbalanced classification model used in operations?

A. Use accuracy alone.

B. Ignore the cost of false negatives.

C. Examine precision, recall, calibration, and operational consequences.

D. Evaluate only training loss.

Correct Answer: C. Examine precision, recall, calibration, and operational consequences.

Question 10

A team wants to send raw payment logs to a generative-AI service for incident summarisation. What is the best first control?

A. Increase the model temperature.

B. Minimise or mask sensitive data and enforce approved data-handling policies.

C. Remove all audit logging.

D. Allow the model to execute remediation directly.

Correct Answer: B. Minimise or mask sensitive data and enforce approved data-handling policies.