Cloud-Native Big Data Platforms

Introduction

Big data systems have moved from organisation-owned data centres to elastic cloud environments. This transition is not simply a change in hardware location. It changes how storage, computation, databases, security, deployment, pricing and operational responsibility are designed. Cloud-native big data platforms use on-demand infrastructure, managed services, automation, distributed architectures and observable operations to process data at large scale.

A cloud-native solution normally separates storage from computation, treats infrastructure as code, uses automated deployment, and designs explicitly for failure. It may combine object storage, managed databases, stream-processing services, serverless query engines, data warehouses and lakehouse components. The central architectural challenge is not choosing the most fashionable service; it is matching workload requirements to appropriate service characteristics while preserving portability, governance and economic control.

This chapter compares major cloud data-platform patterns and develops a practical framework for designing portable solutions with containers and infrastructure automation. It also examines scalability, reliability, cost and vendor lock-in.

Learning Objectives

By the end of this chapter, students should be able to:

1. Compare cloud object storage, managed databases, serverless analytics, data warehouses and lakehouse platforms.
2. Explain the roles of containers, orchestration and infrastructure as code in cloud-native design.
3. Select storage and processing services for batch, streaming, transactional and analytical workloads.
4. Estimate major cost and capacity factors using transparent assumptions.
5. Design for scalability, reliability, security, governance and portability.
6. Evaluate vendor lock-in and make defensible architectural recommendations.

1. Cloud-Native Principles for Big Data

Cloud-native does not mean merely running an application on a cloud provider. It refers to an approach based on several principles:

• Elasticity: resources can increase or decrease in response to demand.
• Automation: provisioning, testing, deployment and recovery are repeatable.
• Service decomposition: storage, ingestion, processing and serving can scale independently.
• Managed operations: the provider operates some infrastructure, allowing the customer to focus on data products.
• Failure awareness: components are expected to fail, so redundancy and recovery are designed in advance.
• Observability: logs, metrics, traces, data-quality indicators and cost measurements support control.
• API-driven management: resources are created and configured through declarative or programmatic interfaces.

Cloud-native design must distinguish scalability from elasticity. Scalability is the ability to handle increased workload by adding resources or improving design. Elasticity is the ability to provision and release resources automatically as workload changes. A system may be scalable but not elastic if operators must resize it manually.

The shared-responsibility model is also fundamental. A provider generally secures the physical facilities, hardware and foundational services, while the customer remains responsible for identities, access policies, data classification, configuration and often operating-system or application security. The exact boundary differs by service model and contract.

2. A Layered Cloud Data-Platform Model

A useful conceptual model contains six layers:

1. Sources: applications, sensors, logs, files, SaaS systems and operational databases.
2. Ingestion: batch transfer, message queues, event streams and change-data capture.
3. Storage: object stores, block or file systems, databases and specialized analytical storage.
4. Processing: distributed batch engines, stream processors, SQL engines and machine-learning services.
5. Serving: dashboards, APIs, operational applications, search systems and feature stores.
6. Governance and operations: identity, metadata, lineage, quality, security, monitoring, deployment and cost management.

These layers are logical rather than necessarily separate products. A managed lakehouse may provide storage, cataloguing and SQL processing together. Nevertheless, separating the concepts helps architects identify dependencies and avoid selecting a single product for every workload.

3. Comparing Cloud Storage and Data Services

3.1 Object storage

Object storage stores data as objects within named containers or buckets. Each object has data, metadata and a key. It is well suited to large files, backups, raw events, images, parquet datasets and durable data-lake storage.

Important characteristics include high durability, virtually unlimited capacity, lifecycle policies and access through APIs. Object storage is usually cheaper per stored byte than database storage, but it is not a general-purpose transactional file system. Operations such as renaming directories, updating individual records and enforcing multi-row transactions are either unsupported or inefficient.

For analytical data, columnar formats such as Parquet or ORC reduce scanned data when queries select only some columns. Partitioning by commonly filtered attributes, such as date or region, can further reduce reads. Poorly chosen partitions, however, create too many small files or cause large partitions to be scanned unnecessarily.

3.2 Managed relational databases

Managed relational database services provide SQL, transactions, indexes, constraints, backups and replication while reducing the administrative burden of patching and hardware management. They are appropriate for operational workloads requiring strong consistency and structured relationships.

They are less suitable as the sole repository for very large historical datasets or high-volume raw events. Horizontal scaling may require sharding, read replicas or a different service. A relational database can therefore serve as the system of record for orders while an object store or warehouse supports historical analysis.

3.3 Managed NoSQL databases

Key-value, document, wide-column and graph databases support access patterns that may not fit relational models. They can provide low-latency access at high request rates, but their schemas, query capabilities and consistency models vary. NoSQL selection should begin with access patterns and consistency requirements rather than with the label “big data.”

3.4 Data warehouses

A cloud data warehouse is a managed analytical system optimised for SQL over structured or semi-structured data. It commonly uses columnar storage, parallel query execution, compression, workload management and separation of compute from storage. Warehouses are effective for governed business intelligence, dimensional models, reporting and interactive aggregations.

Their strengths include mature SQL, security integration, query optimisation and predictable analytical semantics. Limitations may include ingestion or egress charges, proprietary SQL extensions, constraints on unstructured data and cost growth from poorly controlled queries.

3.5 Serverless analytics

Serverless analytics services allow users to submit SQL or processing jobs without managing clusters. Billing often depends on data scanned, execution time, requests or other consumption measures. They are attractive for irregular workloads, exploratory analysis and event-driven processing.

“Serverless” does not mean free of capacity limits, latency variation or operational design. Query engines still require partitioning, file-size management, workload governance, access control and monitoring. A query that scans a complete multi-terabyte table for a small result may be technically successful but economically irresponsible.

3.6 Lakehouse platforms

A lakehouse combines low-cost, flexible data-lake storage with warehouse-like management and analytical capabilities. A typical architecture stores open-format files in object storage and adds a table format that supports transactions, schema evolution, snapshots, metadata and efficient reads.

The lakehouse is useful when an organisation must support raw data, BI, data science and machine learning from a shared foundation. Its difficulties include metadata management, concurrency, small-file compaction, optimisation of open tables, governance across tools and potential dependence on a platform’s proprietary extensions.

4. Workload-Driven Architecture

A sound design starts with workload dimensions:

• data volume and growth rate;
• arrival pattern and burstiness;
• latency target;
• read/write ratio;
• consistency requirement;
• query shape and concurrency;
• retention and deletion obligations;
• security classification;
• recovery-point objective (RPO) and recovery-time objective (RTO);
• budget and portability requirements.

For example, an online payment authorisation service needs low latency, transactional correctness and controlled access. A monthly marketing analysis may tolerate minutes of latency and is better suited to object storage plus a warehouse or serverless engine. A fleet-monitoring system may require streaming ingestion, windowed computation and a time-series serving layer.

A useful separation is between system of record, system of analysis and system of presentation. The system of record preserves authoritative transactions. The analytical system transforms data for exploration and reporting. The presentation system serves dashboards or applications. Replicating data between these roles increases complexity, but forcing all roles into one system usually creates worse performance or governance problems.

5. Containers and Portable Deployment

A container packages an application, its runtime dependencies and configuration conventions into an image that can execute consistently across compatible hosts. Containers improve reproducibility and deployment speed, but they do not automatically make a system portable. Portability is reduced by proprietary APIs, provider-specific identity systems, networking assumptions, storage interfaces and managed-service semantics.

Container orchestration platforms schedule containers, restart failed instances, expose services, manage configuration and support rolling deployment. They are useful for custom ingestion services, stream processors, API layers and data-quality jobs. They may be unnecessary for a simple scheduled SQL transformation that a managed serverless service can execute directly.

Best practices include:

• build minimal images from trusted, maintained base images;
• scan dependencies and images for vulnerabilities;
• run processes as non-root users;
• keep containers stateless where possible;
• store secrets in a secret-management service rather than in images;
• define resource requests and limits;
• emit structured logs and metrics;
• pin versions and test upgrades;
• separate application code from environment configuration.

Containerising a stateful database does not eliminate the complexity of replication, backup, storage performance and recovery. Managed stateful services are often preferable unless the organisation has a clear operational reason to run the database itself.

6. Infrastructure as Code and Automation

Infrastructure as code (IaC) describes networks, identities, storage, compute, databases and policies in version-controlled files. Declarative IaC specifies the desired state; the tool determines actions required to reach that state. Imperative scripts instead specify a sequence of commands. Declarative approaches generally improve reviewability and repeatability, although both may be used.

A robust workflow is:

1. define reusable modules with documented inputs and outputs;
2. review changes through version control;
3. generate a plan before applying changes;
4. apply changes through controlled automation;
5. record state securely and protect it from unauthorised modification;
6. run policy, security and cost checks in the pipeline;
7. test disaster recovery and configuration rollback.

IaC can reduce configuration drift, but it can also reproduce an insecure design rapidly. Code review, least privilege, policy validation and separate development, test and production environments remain essential.

Portability improves when designs use open file formats, standard SQL where practical, container images, Kubernetes-compatible deployment patterns, portable orchestration tools and clear data-access abstractions. Full multi-cloud portability is expensive: different providers expose different identity, networking, observability, pricing and failure semantics. Portability should therefore be treated as a deliberate business option, not an automatic goal.

7. Scalability, Reliability and Performance

7.1 Capacity and throughput

For a simple ingestion pipeline, average required throughput can be estimated as:

where R_avg is average data rate, V is data volume and T is the observation period. Units must be consistent, such as gigabytes per hour. Real systems must also account for peak rate, protocol overhead, replication and processing expansion. If peak traffic is P times the average rate, a basic planning estimate is R_peak = P × R_avg, although measured workload data is preferable.

For a queue or event stream, retained data volume can be approximated by:

where S is retained bytes, r is events per second, t is retention time in seconds and b is average bytes per event. This assumes a stable average rate and excludes replication, indexes and metadata. If there are n replicas, physical storage may approach n × S, subject to the provider’s implementation.

Worked example. A telemetry system receives 20,000 events per second, each averaging 600 bytes, and retains raw events for 72 hours. The retention time is 72 × 3,600 = 259,200 seconds. Thus:

S = 20,000 × 259,200 × 600 = 3,110,400,000,000 bytes, approximately 3.11 TB using decimal units. This is logical event data only; compression, replicas, indexes and file overhead alter physical consumption.

7.2 Reliability concepts

Availability is often expressed as the proportion of time a service is usable:

Here, A is availability, MTBF is mean time between failures and MTTR is mean time to repair or restore. The formula assumes repeated repairable failures and stable average rates; it is a planning model, not a guarantee. If MTBF is 999 hours and MTTR is 1 hour, availability is 999 / 1000 = 0.999, or 99.9%.

Reliability requires redundancy, health checks, automated replacement, backups, tested restoration, multi-zone deployment where appropriate and clear dependency mapping. Replication alone does not guarantee recovery: corrupted data can be replicated, credentials can fail across all replicas and an unavailable control plane may prevent access.

RPO states the maximum tolerable amount of data loss measured in time. RTO states the maximum tolerable time to restore service. A low RPO may require continuous replication, while a low RTO may require warm standby capacity. Both increase cost and operational complexity.

8. Cost and FinOps Reasoning

Cloud cost is multidimensional. A simplified monthly estimate is:

The terms represent storage, compute, API requests, network transfer, managed-service charges and operational costs. This is a planning model; taxes, discounts, support contracts and committed-use pricing may need separate treatment.

For scan-priced analytics, a rough query cost is:

where D_scan is data scanned and p_scan is price per unit of scanned data. The estimate applies only when the service actually bills primarily by scanned data. Partition pruning, column projection and compression can reduce D_scan.

Cost optimisation must not mean reducing reliability or security blindly. Appropriate controls include budgets, query quotas, lifecycle policies, automatic shutdown of non-production resources, rightsizing, storage-tiering, workload scheduling and unit-cost metrics such as cost per processed terabyte or cost per active customer.

9. Governance, Security and Ethics

Big data platforms concentrate sensitive information and therefore require governance by design. Essential controls include:

• identity federation and least-privilege roles;
• encryption in transit and at rest;
• key rotation and separation of duties;
• private networking where justified;
• audit logs that cannot be altered by ordinary operators;
• classification, retention and deletion policies;
• data lineage and catalogues;
• quality checks for completeness, validity and freshness;
• masking or tokenisation of sensitive attributes.

Legal obligations may include privacy, data residency, retention and subject-access or deletion rights. A technically convenient architecture can be unlawful if it transfers personal data to an unauthorised region or retains it indefinitely.

Analytics and machine-learning pipelines introduce fairness, explainability and purpose-limitation concerns. Data collected for service operation should not automatically be reused for behavioural profiling. Access should be based on legitimate purpose, and derived datasets should inherit appropriate sensitivity controls.

10. Vendor Lock-In and Portability

Vendor lock-in occurs when switching providers becomes costly, slow or technically risky. It may arise from proprietary APIs, workflow languages, identity systems, networking, monitoring, data formats, SQL dialects, specialised accelerators, contractual commitments or accumulated operational knowledge.

Lock-in is not always harmful. A proprietary service may provide substantial productivity, reliability or performance advantages. The relevant question is whether the value exceeds the switching risk and whether the organisation has an exit strategy.

A portability assessment should ask:

1. Can data be exported in documented, open formats?
2. Can schemas, metadata and lineage be reconstructed elsewhere?
3. Are transformations expressed in portable SQL or version-controlled code?
4. Are identities and policies mapped to provider-neutral requirements?
5. What are egress costs and expected migration duration?
6. Can the organisation operate the replacement platform?
7. Which features are essential but proprietary?

A practical strategy is selective portability: keep core data in open formats and isolate provider-specific features behind interfaces, while using managed services where their benefits are significant. Multi-cloud deployment should be justified by regulatory, resilience or bargaining requirements; duplicating every service across clouds can double cost and operational burden.

11. Applied Case: Illustrative Retail Telemetry Platform

Background

A fictional national retailer operates 2,000 stores. Point-of-sale transactions require reliable operational processing, while managers need near-real-time inventory dashboards. Historical analysis supports pricing and supply-chain planning. The retailer expects seasonal peaks four times larger than ordinary traffic.

Problem

The existing design writes all events directly into one relational database. Peak periods produce contention between transactions and reporting queries. Storage growth is expensive, data-quality errors are difficult to trace, and recovery procedures have not been tested.

Analysis

The workloads differ. Point-of-sale operations need transactional consistency and low latency. Telemetry and transaction events need durable ingestion and replay. Historical reporting requires columnar analytical scans. Dashboards need a serving layer with predictable response times. The architecture must also protect payment-related data and support deletion or masking obligations.

Approach and Solution

The retailer retains a managed relational database as the operational system of record. Change-data capture publishes approved events to a durable stream. A streaming processor validates schemas, removes or tokenises unnecessary sensitive fields, and writes raw immutable events to object storage in an open columnar format. A table-management layer provides schema evolution and transaction-like table operations for curated data.

A warehouse receives curated sales and inventory tables for governed BI. Serverless queries are reserved for occasional investigation and low-frequency workloads. A dashboard-serving database receives aggregated inventory indicators rather than forcing every dashboard request to scan raw data. Containers run custom validation and enrichment services; IaC defines networks, roles, buckets, streams, policies and deployment environments.

Reliability is improved through multi-zone managed services, replayable streams, object-storage versioning, tested backups and documented RPO and RTO targets. Cost controls include partitioning by event date and store, lifecycle movement of old raw data, query quotas and monitoring of cost per dashboard refresh.

Outcome

The separation removes analytical contention from point-of-sale transactions, permits replay after processing failures and creates a durable historical foundation. The design remains partly provider-specific in identity and managed warehouse features, but open data formats, containerised custom services and exported IaC reduce migration risk.

Learning Points

The case demonstrates that cloud-native architecture begins with workload separation. It also shows that a lakehouse or warehouse is not a replacement for an operational database, and that portability, security, reliability and cost must be designed together rather than added after deployment.

12. Contemporary Developments and Best Practices

Recent platform development includes open table formats, vectorised query engines, data-product architectures, real-time lakehouse ingestion, confidential computing, policy-as-code and AI-assisted operations. AI can help detect anomalous costs, classify data, generate transformation suggestions or summarise operational incidents. It must not replace human review of access policies, data quality or regulatory decisions. Generated code and queries require testing because plausible output can still be insecure or semantically wrong.

Recommended practice is to:

• begin with measurable workload and service-level requirements;
• separate operational, analytical and serving responsibilities;
• prefer open formats and documented interfaces for strategic data;
• automate infrastructure, tests, policy checks and recovery drills;
• measure latency, throughput, freshness, failure rate and unit cost;
• design identity, privacy and retention controls before ingestion;
• use managed services selectively and understand their limits;
• document proprietary dependencies and maintain an exit plan;
• test peak-load behaviour and failure scenarios;
• treat data quality and lineage as production requirements.

Chapter Summary

Cloud-native big data platforms combine elastic infrastructure, managed services, automation and distributed data processing. Object storage provides durable and economical foundations for raw and historical data, while relational and NoSQL databases address operational access patterns. Warehouses offer governed analytical SQL, serverless engines support irregular on-demand analysis, and lakehouses combine flexible object storage with table management and analytical capabilities.

Effective architecture is workload-driven. It separates systems of record, analysis and presentation when their requirements differ. Containers improve reproducibility for custom services, while infrastructure as code makes environments repeatable and reviewable. Scalability requires attention to throughput, burst capacity, partitioning and query design. Reliability depends on redundancy, recovery objectives, backups and tested restoration rather than replication alone.

Cost includes storage, compute, requests, transfer, services and operations. Governance covers identity, encryption, lineage, quality, privacy and retention. Vendor lock-in should be assessed by data formats, APIs, skills, contracts and migration costs. Selective portability—open data and clear interfaces combined with carefully chosen managed services—is often more realistic than complete multi-cloud equivalence.

Key Terms

Cloud-native platform: A platform designed around elastic resources, automation, managed services and failure-aware distributed operation.

Object storage: API-accessed storage of objects and metadata in buckets or containers.

Data warehouse: A managed analytical database optimised for governed SQL workloads.

Serverless analytics: On-demand analytical processing in which users do not manage persistent compute infrastructure.

Lakehouse: A data architecture combining data-lake flexibility with warehouse-like table management and analytics.

Container: A packaged application environment containing code and runtime dependencies.

Orchestration: Automated scheduling, networking, scaling and recovery of containers or jobs.

Infrastructure as code: Version-controlled, machine-readable definition of infrastructure and configuration.

Elasticity: Automatic or rapid adjustment of resources as workload changes.

Scalability: The ability to handle increased workload through additional or more capable resources.

RPO: Recovery-point objective; the maximum acceptable period of data loss.

RTO: Recovery-time objective; the maximum acceptable time to restore service.

MTBF: Mean time between failures.

MTTR: Mean time to repair or restore.

Data lineage: Documentation of data origins, transformations and destinations.

Vendor lock-in: Dependence that makes changing providers costly or difficult.

Partitioning: Organising data into subsets to reduce unnecessary processing.

Change-data capture: Recording inserts, updates and deletes from a source system for downstream processing.

FinOps: Financial management practices for controlling and improving cloud economics.

Review Questions

Short Answer Questions

1. What distinguishes elasticity from scalability?
2. Why is object storage useful for data lakes?
3. State two reasons for separating an operational database from an analytical warehouse.
4. What do RPO and RTO measure?
5. Identify three sources of cloud vendor lock-in.

Long Answer Questions

1. Compare object storage, data warehouses and lakehouse platforms for an enterprise analytics programme.
2. Explain how containers and infrastructure as code contribute to reproducible cloud deployments.
3. Discuss the reliability controls required for a replayable streaming data platform.
4. Explain how partitioning, columnar formats and query governance influence analytical cost.
5. Evaluate the claim that serverless services eliminate operational responsibility.

Analytical / Application Questions

1. A platform receives 5,000 events per second, each averaging 800 bytes, and retains them for 48 hours. Estimate logical retained data volume using S = r × t × b, state your assumptions, and identify at least three additional storage factors.
2. Design a cloud-native architecture for a healthcare organisation that requires transactional applications, daily reporting and near-real-time alerts. Justify storage, processing, security and governance choices.
3. An organisation is considering a proprietary warehouse that reduces development time but charges high data-egress fees. Develop a decision framework addressing performance, reliability, cost, portability and regulatory risk.

Multiple Choice Questions with Answers

Question 1

Which workload is generally best suited to cloud object storage?

A. Multi-row transactional order updates

B. Durable storage of raw historical files

C. Low-latency indexed customer lookups

D. Strict serialisation across bank accounts

Correct Answer: B. Durable storage of raw historical files

Question 2

What is the main purpose of infrastructure as code?

A. To eliminate the need for security reviews

B. To define and reproduce infrastructure through version-controlled specifications

C. To guarantee that every cloud provider has identical services

D. To replace all application testing

Correct Answer: B. To define and reproduce infrastructure through version-controlled specifications

Question 3

A system receives 1,000 events per second, each 500 bytes, for 3,600 seconds. What is the logical event volume?

A. 1.8 MB

B. 18 MB

C. 1.8 GB

D. 18 GB

Correct Answer: C. 1.8 GB

Question 4

Which feature most directly reduces data scanned by a suitable analytical query?

A. Partition pruning

B. Increasing dashboard colours

C. Adding unused columns

D. Disabling compression

Correct Answer: A. Partition pruning

Question 5

What does RPO describe?

A. Maximum acceptable service restoration time

B. Maximum acceptable amount of data loss measured in time

C. Average time between software releases

D. Maximum number of concurrent users

Correct Answer: B. Maximum acceptable amount of data loss measured in time

Question 6

Which statement about containers is correct?

A. Containers automatically provide database backup and recovery

B. Containers package applications and dependencies for consistent execution

C. Containers remove all provider-specific dependencies

D. Containers can safely store unencrypted secrets in images

Correct Answer: B. Containers package applications and dependencies for consistent execution

Question 7

Which platform is primarily optimised for governed SQL business intelligence?

A. Data warehouse

B. Message queue

C. Container registry

D. Object metadata service

Correct Answer: A. Data warehouse

Question 8

If MTBF is 999 hours and MTTR is 1 hour, the availability estimate using MTBF ÷ (MTBF + MTTR) is:

A. 90%

B. 99%

C. 99.9%

D. 99.99%

Correct Answer: C. 99.9%

Question 9

Which is an example of selective portability?

A. Running every service identically on five clouds

B. Avoiding all managed services

C. Using open data formats while accepting some provider-specific managed features

D. Storing all data only in proprietary binary formats

Correct Answer: C. Using open data formats while accepting some provider-specific managed features

Question 10

Why should a cloud data platform separate systems of record and analysis when appropriate?

A. To ensure analytical queries can never be performed

B. Because operational transactions and analytical scans often have different performance and consistency requirements

C. Because object storage cannot store data

D. To remove the need for data governance

Correct Answer: B. Because operational transactions and analytical scans often have different performance and consistency requirements