Cloud Big Data Platforms and DataOps/MLOps

Introduction and context

Cloud computing has changed big data from a capital-intensive infrastructure problem into a programmable service-design problem. Organisations can now store petabytes in object storage, allocate distributed computing when needed, expose analytical workloads through SQL, deploy machine-learning models through managed endpoints, and automate the movement from code to production. The resulting flexibility is valuable, but it also creates new responsibilities: architecture must control cost, security, reliability, data quality, reproducibility, and operational risk.

This chapter examines cloud-native platforms and the operating practices that make them dependable. It distinguishes storage from computation, infrastructure from managed services, and DataOps from MLOps. The emphasis is not on a particular vendor but on transferable design principles applicable to major cloud ecosystems.

Learning objectives

By the end of this chapter, you should be able to:

1. Evaluate object, block, file and analytical storage for big data workloads.
2. Compare virtual machines, containers, clusters, serverless functions and managed analytical services.
3. Explain container images, orchestration, elasticity and service reliability.
4. Design a DataOps/MLOps workflow involving versioning, testing, deployment, monitoring and observability.
5. Apply cost-performance reasoning to storage, compute, networking and model serving.
6. Identify governance, ethical, security and regulatory risks in cloud analytics.

1. The cloud-native analytical architecture

A cloud-native analytics platform separates several layers that were often purchased and operated as one system in a traditional data centre:

• Data sources: applications, sensors, transactions, logs, external feeds and files.
• Ingestion: batch transfer, event streaming, change-data capture and application programming interfaces (APIs).
• Storage: durable object stores, relational databases, data warehouses, data lakes and lakehouses.
• Processing: distributed batch engines, stream processors, SQL engines, notebooks and machine-learning runtimes.
• Serving: dashboards, semantic layers, APIs, feature stores and model endpoints.
• Control plane: identity, policy, metadata, orchestration, deployment, monitoring and billing.

The key architectural idea is decoupling. Storage can persist independently of a compute cluster; compute can be scaled for a workload and then released. This is called elasticity. It differs from simple scalability: scalability is the ability to handle greater demand, whereas elasticity is the ability to acquire and release resources in response to changing demand.

A typical data path is:

Source → ingestion → raw storage → validation and transformation → curated storage → analytical processing → dashboard or model → monitored outcome.

A platform should also preserve the reverse path: alerts, lineage, quality results, model predictions and business outcomes must feed back into operations and improvement.

2. Cloud-native storage

2.1 Object storage

Object storage stores objects consisting of data, metadata and a key within a bucket or similar namespace. It is highly durable, broadly accessible and well suited to unstructured or semi-structured data such as JSON, CSV, Parquet files, images, logs and model artefacts.

Its strengths include low cost per stored byte, lifecycle policies, versioning, replication and integration with analytical engines. Its limitations include higher access latency than local disks, restrictions on in-place updates, and the need for application-level conventions for directories, schemas and transactions. Object storage should therefore be treated as a durable data foundation, not automatically as a database.

Columnar formats such as Parquet and ORC are important because analytical queries often read only selected columns. Compression and partitioning can substantially reduce bytes scanned. Partitioning by date, region or another frequently filtered attribute can improve performance, but excessive partitioning creates many small files and metadata overhead.

2.2 Block and file storage

Block storage presents volumes to virtual machines. It provides predictable low-latency reads and writes and is useful for databases, transactional workloads and local working space. File storage presents a shared hierarchical file system and is useful when multiple compute instances require common file semantics. Both generally cost more per unit of capacity than object storage and require more active capacity planning.

2.3 Warehouses, lakes and lakehouses

A data warehouse provides governed, structured, SQL-oriented analytical storage. It commonly uses columnar execution, query optimisation and separate compute from storage. A data lake stores data in relatively open formats, often with varied structure and an emphasis on flexibility. A lakehouse combines lake storage with warehouse-like table management, transactions, schema enforcement and governance.

The choice is not ideological. A warehouse is often effective for stable, high-value reporting; a lake is useful for raw data, exploration and diverse formats; and a lakehouse can reduce duplication when the same data supports BI, data science and machine learning. The design must address table transactions, concurrent writes, schema evolution, cataloguing and access control.

2.4 Storage selection criteria

Storage should be evaluated using workload evidence rather than brand familiarity. Important criteria include:

A useful storage policy combines hot, warm and cold tiers. Frequently accessed data remains in a high-performance tier; older data is moved to cheaper archival storage according to retention and legal requirements. Lifecycle automation must be tested because premature archival can increase retrieval cost and delay analysis.

3. Computing models and managed services

Virtual machines and clusters

Virtual machines provide operating-system-level control, custom drivers and predictable environments. They are appropriate when software needs privileged access, specialised hardware or long-running services. However, teams must patch operating systems, manage capacity and respond to failures.

Distributed processing clusters use multiple machines for parallel computation. A large job is divided into tasks, scheduled across workers and often protected by replication or recomputation. Cluster systems are powerful for large transformations but introduce scheduling, dependency, shuffle, tuning and capacity-management challenges.

Containers

A container packages an application and its user-space dependencies into an image. The host kernel is shared, so containers are generally lighter and faster to start than virtual machines, although they are not a complete security boundary by themselves. A good image is small, reproducible, scanned for vulnerabilities and identified by an immutable digest rather than an ambiguous mutable tag.

A container image usually contains application code, libraries, configuration defaults and a startup command. Secrets should not be baked into images; they should be injected at runtime through a secrets-management facility. Images should be built using automated pipelines, signed where appropriate, and promoted between environments rather than rebuilt differently in each environment.

Orchestration

An orchestrator schedules containers, restarts failed instances, provides service discovery, manages configuration, supports rolling updates and can scale workloads. A desired-state model is central: the operator specifies the desired number and type of workloads, while the platform continuously attempts to reconcile actual state with desired state.

Important orchestration concepts include:

• Pod or task: the smallest deployable workload unit in a platform.
• Service discovery: a stable mechanism for locating changing instances.
• Readiness: whether a workload can receive traffic.
• Liveness: whether a workload should be restarted.
• Horizontal scaling: changing the number of instances.
• Vertical scaling: changing resources assigned to an instance.
• Rolling deployment: replacing instances gradually to reduce interruption.
• Namespace or project boundary: logical separation for teams and policies.

Autoscaling must be linked to a meaningful signal such as queue depth, request latency or CPU utilisation. Scaling only on CPU can fail when the bottleneck is memory, I/O, external rate limits or a slow database.

Serverless computing

Serverless functions and serverless analytical services abstract away server provisioning. Users submit code or queries and pay according to execution, requests, data processed or other service-specific measures. Serverless is attractive for irregular workloads, event handling, rapid experimentation and low operational overhead.

Limitations include startup latency, execution-duration limits, concurrency quotas, vendor-specific interfaces, difficult local reproduction and potentially surprising cost under high volume. Serverless is not literally server-free; responsibility for application correctness, identity, data protection and cost control remains with the organisation.

Managed analytics services

Managed services may include data warehouses, streaming platforms, orchestration services, notebook environments, feature stores and model endpoints. They reduce undifferentiated operations, but they do not eliminate architecture. A managed service still requires workload sizing, access control, data contracts, monitoring, backup decisions and exit planning.

Evaluation should consider functional fit, performance, reliability objectives, integration, skills, portability, commercial terms and regulatory location. A service that is technically excellent but difficult to govern or excessively expensive at the organisation’s scale is not an effective choice.

4. DataOps: making data delivery reliable

DataOps is a collaborative and automated approach to developing, testing, deploying and operating data products. It applies principles from DevOps to data while recognising data-specific problems: changing schemas, late-arriving events, duplicates, silent quality degradation, lineage and ambiguous business definitions.

Data contracts and quality

A data contract states the expected schema, semantic meaning, ownership, freshness, valid ranges and compatibility rules for a dataset or event. It creates an explicit agreement between producers and consumers. A useful contract might state that customer_id is non-null, unique within a customer dimension, and represented as a string; that event time is in UTC; and that a transaction amount is non-negative and expressed in a specified currency.

Quality tests can include:

• completeness: required fields are present;
• validity: values satisfy type and domain rules;
• uniqueness: keys do not unexpectedly repeat;
• consistency: related tables agree;
• timeliness: data arrives within the service objective;
• distribution checks: values have not shifted abnormally.

A quality failure should produce an observable event and an operational response, not merely a red dashboard. Possible responses include quarantining records, stopping publication, routing to a dead-letter store, or publishing with an explicit warning.

Versioning and reproducibility

Code, infrastructure definitions, schemas, transformation logic, dependencies, configuration and selected data artefacts should be versioned. Git records code and configuration; a data or model registry records larger artefacts and metadata. A reproducible run should identify the code revision, input snapshot or time window, environment, parameters and output location.

Data versioning is not always a byte-for-byte copy of every dataset. It may use immutable raw data, table snapshots, change logs, or content-addressed artefacts, subject to privacy and retention constraints. Reproducibility must also respect deletion obligations: legal erasure may require removing or reprocessing data even when historical snapshots exist.

Continuous integration and delivery

Continuous integration (CI) automatically checks changes when they are proposed. Typical tests include unit tests for transformations, schema tests, SQL linting, dependency checks, security scans and small integration tests against representative data.

Continuous delivery (CD) packages an approved change and promotes it through development, staging and production. Deployment strategies include:

• Blue-green: two environments exist; traffic switches from the old to the new version.
• Canary: a small percentage of traffic uses the new version before expansion.
• Rolling: instances are replaced incrementally.
• Shadow: the new version receives copied inputs but does not affect users.

Database and table changes require special care. A backward-compatible change, such as adding a nullable column, can often be deployed before consumers change. A breaking change should use a migration plan, compatibility period and explicit deprecation.

5. MLOps: operating the machine-learning lifecycle

MLOps extends DataOps and DevOps to models whose behaviour depends on data, features, code, parameters and the environment. The lifecycle includes problem definition, data preparation, training, validation, registration, deployment, monitoring, retraining and retirement.

A model registry should record the model version, training-data reference, feature definitions, metrics, responsible owner, approval status, dependencies and intended use. Promotion should be based on predefined acceptance criteria rather than an informal notebook result.

Validation and deployment

Model validation includes technical performance and operational suitability. Depending on the task, metrics may include accuracy, precision, recall, F1 score, calibration, area under a relevant curve, forecast error or ranking quality. The metric must match the decision cost. For example, a fraud system may value recall differently from a marketing recommendation system.

Before deployment, teams should test:

• input schema and feature availability;
• missing-value and outlier behaviour;
• latency and throughput;
• fairness across relevant groups;
• robustness to distribution changes;
• privacy and security controls;
• rollback and fail-safe behaviour.

Deployment may be batch scoring, an online endpoint, an embedded model, or an edge deployment. Batch inference is often cheaper and simpler when decisions do not require immediate responses. Online inference is justified when latency has business value and the organisation can operate a reliable serving path.

Monitoring models

Model monitoring includes system, data and outcome dimensions. System monitoring measures latency, errors, saturation and availability. Data monitoring measures missingness, schema changes and feature distributions. Outcome monitoring measures performance when labels become available. Data drift describes a change in input distributions; concept drift describes a change in the relationship between inputs and the target. Drift is a signal for investigation, not automatic proof that retraining will improve performance.

A production model should have a rollback or fallback plan. A fallback may be the previous approved model, a rules-based process, human review or a safe default. Retraining should be triggered by evidence and governance, not by an unexamined calendar alone.

6. Observability, reliability and service objectives

Monitoring reports known measurements; observability supports investigation of unknown failure modes. The three widely used telemetry types are:

• Metrics: numeric time series such as request rate, error rate and queue depth.
• Logs: timestamped event records with contextual fields.
• Traces: linked spans that show a request’s path across services.

Correlation identifiers allow an analyst to connect an ingestion event, transformation run, query and model prediction. Logs should be structured, time-synchronised and scrubbed of secrets and unnecessary personal data.

A service-level indicator (SLI) measures an aspect of service behaviour, such as successful requests divided by valid requests. A service-level objective (SLO) is the target for that indicator, such as 99.9% successful requests over a defined period. A service-level agreement (SLA) is a contractual commitment, often with remedies. These terms should not be treated as interchangeable.

For an availability SLI, a common formulation is:

More precisely, when measured over a fixed observation window, availability is usually:

Both time quantities must use the same unit, and the definition of downtime must be agreed in advance. If a service is scheduled for 720 hours and unavailable for 1.44 hours, availability is (720 − 1.44) ÷ 720 = 0.998 = 99.8%. The result does not imply that every request succeeded; request-based and time-based indicators measure different properties.

Incident response should include alert thresholds, ownership, escalation, runbooks, post-incident review and corrective actions. Alert fatigue is a design failure: alerts should be actionable and tied to user or business impact.

7. Cost-performance optimisation

Cloud economics require measurement at the workload level. Total cost may include storage, compute, requests, data transfer, managed-service units, observability, support and idle resources. A simple workload-cost model is:

This is an accounting decomposition, not a universal provider billing equation. Each component must be defined using the provider’s pricing dimensions and the selected time period.

For a compute task, an approximate cost is:

This assumes a stable hourly price and excludes discounts, storage, networking and ancillary charges. For example, 20 instances used for 3 hours at $0.40 per instance-hour cost approximately 20 × 3 × $0.40 = $24. If the same work is completed by 10 instances in 4 hours, the cost is $16, although elapsed time increases by one hour. The cheaper choice is not automatically better: business value, SLOs and opportunity cost also matter.

Performance optimisation techniques include partition pruning, columnar formats, compression, appropriate file sizes, caching, efficient joins, autoscaling, workload isolation and query-result reuse. Cost optimisation techniques include lifecycle policies, rightsizing, scheduling non-production resources, reserved or committed capacity where demand is predictable, quotas and budgets.

A useful decision is often multi-objective. A design can be compared by a weighted score such as:

where C, P and R are normalised cost, performance and reliability measures, and w_c, w_p and w_r are non-negative organisational weights summing to 1. This score is a decision aid, not an objective truth; normalisation and weights must be documented, and hard constraints such as legal compliance should be treated as gates rather than traded away.

8. Governance, security and responsible use

Cloud analytics concentrates valuable and sensitive information. Governance must therefore be designed into the platform. Core controls include least-privilege identity, multi-factor authentication, encryption in transit and at rest, key management, network segmentation, private endpoints where appropriate, audit logs, vulnerability management and tested recovery procedures.

Data classification should distinguish public, internal, confidential and highly restricted information. Access should be granted to roles or attributes, reviewed periodically and removed when no longer required. Tokenisation, masking and row- or column-level policies can reduce exposure. Backup copies and logs must receive the same privacy consideration as primary data.

Regulatory duties may include lawful processing, purpose limitation, retention control, data-subject rights, breach response, cross-border transfer rules and sector-specific requirements. A cloud provider’s compliance certification does not transfer the organisation’s accountability for its use of data.

MLOps adds risks of discriminatory outcomes, opaque decisions, proxy variables, feedback loops and inappropriate automation. Human oversight should be proportionate to harm. Documentation should state intended use, limitations, known performance differences, monitoring responsibilities and a route for contesting consequential decisions.

9. Illustrative case: a regional retailer’s real-time analytics platform

Background

A fictional regional retailer operates 350 stores and an online channel. Its transaction data is in a warehouse, clickstream data is in application logs, and inventory updates arrive from store systems. The company wants near-real-time stockout alerts and a demand-forecasting model.

Problem

The existing nightly batch process delays alerts until the next day. It also creates duplicate customer events, has no formal schema ownership and runs on an oversized cluster continuously. Analysts cannot reproduce a forecast because training data and code are not versioned together.

Analysis

The business requires alerts within five minutes, but demand forecasts can be refreshed hourly. Raw events need durable retention, while dashboards need governed, curated tables. The platform must tolerate late events and store-level connectivity failures. Cost analysis shows that a continuously running cluster is poorly matched to irregular demand.

Approach and solution

The retailer adopts the following design:

1. Store and web events are ingested into a managed streaming service and copied unchanged to versioned object storage.
2. A schema registry and data contract define event names, identifiers, event time and currency.
3. Stream processing performs validation, deduplication and enrichment. Invalid records go to a quarantined location with an alert.
4. Curated tables are stored in an open columnar format and exposed through a managed SQL engine.
5. Serverless queries support intermittent analyst workloads; scheduled jobs use ephemeral managed compute rather than an always-on cluster.
6. A containerised feature pipeline is built and tested in CI. Model artefacts and metadata are registered, and a batch model produces hourly forecasts.
7. A canary release compares the new forecasting model with the approved model. Monitoring tracks data freshness, forecast error, endpoint health and store-level performance.
8. Dashboards display cost by workload, bytes scanned, late-event rates and alert latency.

Outcome

In this illustrative scenario, alerts become available within the target window, analysts can reproduce model runs from recorded inputs and code revisions, and non-production compute is shut down automatically. The platform does not eliminate all failures: store connectivity and changing consumer behaviour remain sources of uncertainty. However, failures become visible, bounded and recoverable.

Learning points

The case demonstrates that cloud adoption alone is not transformation. Benefits came from separating storage and compute, choosing processing modes by latency need, establishing contracts, automating tests, versioning artefacts and monitoring business outcomes. It also shows why cost, reliability and governance must be designed together.

10. Best-practice design checklist

• Begin with workload requirements, data sensitivity and service objectives.
• Prefer open, documented data formats when portability matters.
• Keep raw data immutable where lawful, and create governed curated layers.
• Use infrastructure as code and peer-reviewed change control.
• Build, scan, sign and promote immutable container images.
• Keep secrets outside source code and images.
• Test schemas, transformations, security and failure recovery.
• Separate deployment success from business outcome success.
• Instrument metrics, logs and traces before production launch.
• Monitor cost as an operational metric, not only as a monthly surprise.
• Design rollback, quarantine and human escalation paths.
• Review access, retention, model fairness and regulatory obligations.

Contemporary developments

Current platforms increasingly support open table formats, unified batch and streaming pipelines, confidential computing, specialised accelerators, policy-aware catalogs and natural-language interfaces. Generative AI can assist with SQL, code and incident analysis, but generated artefacts require testing, review and protection against data leakage. Platform engineering is also becoming more prominent: internal teams provide paved, self-service paths that standardise security, observability and deployment without preventing justified exceptions.

These developments do not invalidate foundational principles. Greater automation increases the need for provenance, evaluation, access control and human accountability. A sophisticated platform is successful only when it produces trustworthy decisions at an acceptable cost.

Chapter Summary

Cloud-native big data platforms separate durable storage from elastic computation and provide multiple execution models, including virtual machines, clusters, containers, orchestration, serverless functions and managed analytical services. Object storage offers durable, economical foundation storage, while warehouses, lakes and lakehouses serve different combinations of structure, governance and flexibility.

DataOps applies automated engineering discipline to data pipelines through contracts, quality tests, versioning, CI/CD and observable operations. MLOps extends these practices to models by governing training data, features, model artefacts, validation, deployment, drift monitoring, rollback and retirement. Containers improve packaging consistency, while orchestration manages desired state, service discovery, health and scaling.

Reliable cloud analytics requires metrics, logs, traces, service objectives, incident response and cost measurement. Optimisation must balance cost, performance, reliability, compliance and business value. Security, privacy, fairness and regulatory accountability remain organisational responsibilities even when infrastructure is managed by a provider.

Key Terms

Cloud-native architecture: A design that uses programmable, elastic and managed cloud capabilities while automating operations.

Object storage: Durable key-based storage for objects, metadata and large-scale unstructured or semi-structured data.

Data warehouse: Governed, structured analytical storage optimised for SQL and reporting.

Data lake: Flexible storage repository supporting varied data formats and analytical uses.

Lakehouse: A platform combining lake flexibility with warehouse-style transactions and governance.

Elasticity: The ability to acquire and release resources as workload demand changes.

Container: An isolated application process packaged with its user-space dependencies.

Orchestration: Automated scheduling, scaling, networking and lifecycle management of workloads.

Serverless computing: A consumption-oriented model in which infrastructure provisioning is abstracted from the user.

DataOps: Automated, collaborative practices for reliable data-product delivery and operation.

Data contract: An explicit agreement about a dataset’s schema, meaning, quality and service expectations.

Data quality: The degree to which data is complete, valid, consistent, unique and timely for its purpose.

MLOps: Practices for developing, deploying, monitoring and governing machine-learning systems.

Model registry: A controlled repository of model versions and their metadata.

Data drift: Change in the distribution of model inputs over time.

Concept drift: Change in the relationship between inputs and the prediction target.

Observability: The ability to understand internal system behaviour from emitted telemetry.

Service-level objective: A target level for a measured service indicator.

Infrastructure as code: Declarative, version-controlled management of infrastructure resources.

Canary deployment: Gradual release of a change to a small portion of traffic before wider rollout.

Review Questions

5 Short Answer Questions

1. What is the principal difference between object storage and block storage?
2. Define elasticity and explain how it differs from scalability.
3. What purposes do readiness and liveness checks serve in orchestration?
4. Why should secrets not be embedded in container images?
5. Distinguish data drift from concept drift.

5 Long Answer Questions

1. Compare warehouses, data lakes and lakehouses for a retailer supporting BI, data science and machine learning.
2. Explain how DataOps practices improve the reliability and reproducibility of data pipelines.
3. Describe an MLOps lifecycle from training-data preparation through model retirement.
4. Analyse the roles of metrics, logs, traces, SLIs, SLOs and incident response in a production platform.
5. Discuss how security, privacy, governance and fairness should influence cloud analytics architecture.

3 Analytical / Application Questions

1. An organisation has irregular nightly workloads and an always-on cluster. Propose a revised architecture and identify the evidence you would collect to evaluate cost and performance.
2. A model’s input distribution changes substantially, but its measured business outcome remains stable. Explain possible interpretations and recommend an investigation plan.
3. Design a deployment and rollback strategy for a real-time fraud model where false negatives are costly and prediction latency must remain low.

Multiple Choice Questions with Answers

Question 1

Which storage type is generally most appropriate as a durable foundation for large collections of Parquet files and logs?

A. Object storage

B. Block storage attached to one virtual machine

C. A local temporary disk

D. A container image registry

Correct Answer: A. Object storage

Question 2

What is the main purpose of a data contract?

A. To guarantee that a cloud provider will reduce prices

B. To define expected schema, meaning, quality and ownership for data

C. To replace all data-quality tests

D. To encrypt every data record automatically

Correct Answer: B. To define expected schema, meaning, quality and ownership for data

Question 3

Which deployment strategy sends a small proportion of production traffic to a new version before wider release?

A. Blue-green deployment

B. Canary deployment

C. Shadow deletion

D. Cold archival

Correct Answer: B. Canary deployment

Question 4

A service is scheduled for 720 hours and has 1.44 hours of downtime. What is its time-based availability?

A. 98.0%

B. 99.0%

C. 99.8%

D. 99.98%

Correct Answer: C. 99.8%

Question 5

Which telemetry type is best suited to showing the path of one request across several services?

A. A trace

B. A storage lifecycle rule

C. A schema registry

D. A billing budget

Correct Answer: A. A trace

Question 6

Which statement about serverless computing is correct?

A. It eliminates all operational responsibilities

B. It always costs less than a continuously running service

C. It abstracts much infrastructure provisioning from the user

D. It cannot be used for event-driven workloads

Correct Answer: C. It abstracts much infrastructure provisioning from the user

Question 7

What does concept drift describe?

A. A change in container image size

B. A change in the relationship between inputs and the target

C. A temporary increase in storage capacity

D. A change in cloud billing currency

Correct Answer: B. A change in the relationship between inputs and the target

Question 8

Which practice best supports reproducibility of a machine-learning run?

A. Recording only the final accuracy

B. Keeping code, input reference, environment and parameters identifiable

C. Rebuilding the environment manually for each run

D. Deleting intermediate metadata after deployment

Correct Answer: B. Keeping code, input reference, environment and parameters identifiable

Question 9

A compute task uses 10 instances for 4 hours at $0.40 per instance-hour. Ignoring other charges, what is the approximate cost?

A. $4

B. $16

C. $40

D. $160

Correct Answer: B. $16

Question 10

Why is a model drift alert not, by itself, proof that retraining will improve the model?

A. Drift is impossible to measure

B. Drift is a signal requiring investigation of data and outcomes

C. Retraining can never change model performance

D. Production models do not use input data

Correct Answer: B. Drift is a signal requiring investigation of data and outcomes