Scalable, Cloud and Real-Time Data Platforms

Introduction

Modern organisations generate data from transactional systems, mobile applications, sensors, machines, social platforms, web interactions and business processes. The volume, velocity and variety of this data frequently exceed the practical limits of a single-server relational database. Data mining and machine learning therefore depend not only on algorithms, but also on platforms capable of storing, processing and governing data at scale.

A scalable data platform distributes storage and computation across multiple machines or services. Contemporary platforms may combine distributed file systems, parallel processing engines, NoSQL databases, event-streaming systems, cloud warehouses, data lakes and lakehouses. The appropriate architecture depends on latency requirements, data structure, workload patterns, consistency needs, regulatory constraints and cost.

This chapter compares major platform families and explains how they support batch analytics, interactive querying, machine learning and real-time decision-making.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain distributed storage and parallel processing principles.

2. Compare Hadoop and Spark ecosystems.

3. Select appropriate NoSQL models for common applications.

4. Describe streaming architectures and event-time processing.

5. Distinguish cloud data warehouses, data lakes and lakehouses.

6. Analyse modern data-stack architectures and their trade-offs.

7. Evaluate scalability, reliability, governance, cost and latency.

8. Propose a suitable platform for an applied data-mining problem.

1. Foundations of Scalable Data Platforms

1.1 Vertical and horizontal scaling

Vertical scaling increases the capacity of one server by adding CPU, memory, storage or network capability. It is comparatively simple but eventually encounters hardware, cost and availability limits. Horizontal scaling adds servers and distributes data or computation among them. It can provide greater aggregate capacity, although it introduces coordination, partitioning, failure-handling and consistency challenges.

A platform is described as scalable when its useful capacity can increase as resources are added. Scalability is not identical to speed. A system may process a small dataset quickly but scale poorly because coordination or data movement grows disproportionately with dataset size.

1.2 Distributed storage

Distributed storage divides data into partitions, often called shards or blocks, and places copies on multiple nodes. Partitioning enables parallel access; replication improves availability and durability. A distributed system must decide how to handle node failure, network delay, inconsistent replicas and rebalancing when nodes are added or removed.

Important design dimensions include:

• Partitioning: assigning records to nodes, commonly by hash, range or geographic region.
• Replication: maintaining multiple copies of data.
• Consistency: defining what readers can observe after updates.
• Durability: protecting data against loss.
• Availability: continuing to serve requests during failures.
• Locality: processing data near where it is stored to reduce network transfer.

The CAP theorem is a useful conceptual model: in the presence of a network partition, a distributed system must trade off strong consistency and availability. CAP does not state that a system can provide only two properties under all circumstances; rather, it describes the unavoidable choice during partition conditions. Real systems often offer tunable or application-specific consistency.

1.3 Distributed processing

Distributed processing divides a job into tasks that execute concurrently. A typical pipeline contains data ingestion, partitioning, transformation, aggregation and output. Parallel execution can reduce elapsed time, but the benefit is limited by sequential work, communication and coordination.

A useful interpretation comes from Amdahl’s law. If a fraction \(s\) of a workload is inherently sequential and the remaining fraction \(1-s\) can use \(p\) parallel workers, the ideal speedup is:

\[

S(p)=\frac{1}{s+(1-s)/p}

\]

Here, \(S(p)\) is speedup relative to one worker, \(s\) and \(1-s\) are proportions of execution time, and \(p\) is the number of workers. The equation assumes balanced work, negligible communication overhead and independent parallel tasks. Real systems achieve less because of data skew, scheduling, serialization, network transfer and failures.

For example, if 10% of a job is sequential and 8 workers process the parallel part, the ideal speedup is:

\[

S(8)=1/(0.10+0.90/8)=1/0.2125\approx4.71

\]

Adding workers beyond this point produces diminishing returns unless the sequential component is reduced.

2. The Hadoop Ecosystem

Hadoop provided an influential open-source model for storing and processing large datasets on clusters of commodity hardware.

2.1 HDFS

The Hadoop Distributed File System (HDFS) stores files as blocks distributed across DataNodes, while NameNode services maintain namespace and block metadata. Files are replicated so that failure of a node does not necessarily cause data loss. HDFS is designed mainly for large, sequential reads and writes rather than numerous small, low-latency updates.

Its strengths include high-throughput batch access, fault tolerance and economical storage. Its limitations include metadata overhead for many small files, relatively high latency for individual queries and less suitability for frequent record-level mutation.

2.2 MapReduce

MapReduce expresses a computation through two central stages. A map function transforms input records into intermediate key-value pairs. A shuffle partitions and transfers intermediate values by key. A reduce function aggregates or processes all values associated with a key.

For word counting, a mapper emits `(word, 1)` for each token. The shuffle groups identical words, and a reducer sums the associated values. The model is robust because intermediate results can be materialised and failed tasks can be rerun. However, repeated disk writes make multi-stage iterative algorithms, such as many machine-learning methods, relatively slow.

2.3 YARN and the wider ecosystem

YARN separates cluster resource management from processing models. This allowed MapReduce and other engines to share a cluster. Related Hadoop technologies historically included Hive for SQL-like batch querying, Pig for data flows, HBase for column-family storage, Sqoop for relational import and Oozie for workflow coordination. Many of these tools have been supplemented or replaced by cloud-native services and newer engines, but their design concepts remain educationally important.

3. Apache Spark and Parallel Analytics

Apache Spark is a general distributed processing engine supporting batch processing, SQL, streaming and machine learning. It commonly uses memory to accelerate repeated operations, while retaining fault tolerance through lineage and recomputation. Spark can also use disk when data exceeds available memory.

3.1 Transformations and actions

A Spark transformation, such as `filter`, `select` or `join`, describes a new dataset. An action, such as `count`, `collect` or `write`, triggers execution. Spark builds a directed acyclic graph (DAG) of operations and optimises the execution plan. Narrow transformations can often be performed within a partition; wide transformations, such as many joins and aggregations, require a shuffle.

Shuffles are expensive because they involve network transfer, serialization and possible disk spill. Good design therefore includes filtering early, selecting only required columns, using suitable partitioning and avoiding unnecessary wide operations.

Spark provides DataFrame and SQL interfaces that allow a query optimiser to improve execution. RDDs provide lower-level control but generally require more programming effort. Structured Streaming extends the DataFrame model to continuously arriving data.

3.2 Hadoop and Spark comparison

Hadoop MapReduce is strongly associated with disk-oriented batch computation and explicit map and reduce stages. Spark provides a broader execution model, DAG optimisation and efficient support for iterative workloads. Spark is not a replacement for all Hadoop components: it may run with different storage systems and resource managers, including cloud object storage and Kubernetes.

The correct choice depends on workload. Large, simple, durable batch pipelines may be adequately served by a disk-oriented engine. Interactive analytics, iterative machine learning and mixed batch-streaming workloads often benefit from Spark. Neither framework automatically solves data quality, governance or poor partitioning.

4. NoSQL Data Stores

NoSQL systems are non-relational databases designed for particular access patterns, scale requirements or data models. “NoSQL” does not mean that SQL is impossible, nor does it mean that relational systems are obsolete.

4.1 Key-value stores

A key-value store associates a unique key with an opaque value. It is appropriate for sessions, shopping carts, caching and simple lookups. It offers high performance when access is primarily by key but is weak for ad hoc relationships and complex joins.

4.2 Document stores

Document databases store semi-structured documents, commonly JSON-like objects. They are useful when records have variable attributes or when an aggregate should be read and written together, as in product catalogues or content management. Embedding can improve read performance, but excessive duplication complicates updates.

4.3 Wide-column stores

Wide-column systems organise data by partition key and clustering columns. They are suitable for large-scale time-series, event and operational workloads with known query patterns. Data modelling is query-driven: a table may be designed specifically for each important access path.

4.4 Graph databases

Graph databases represent vertices, edges and properties. They support relationship-intensive problems such as fraud detection, recommendation, identity resolution and network analysis. They are less suitable when the primary workload is large sequential aggregation over independent records.

4.5 NoSQL trade-offs

NoSQL systems often scale horizontally and offer flexible schemas, but they may provide limited joins, eventual consistency or restricted transactions. A design should begin with required queries, transaction boundaries, failure behaviour and compliance obligations. Selecting a database solely because it is popular is poor engineering practice.

5. Streaming Data and Real-Time Analytics

Batch processing operates on bounded datasets. Stream processing handles unbounded sequences of events. Examples include payment authorisations, sensor measurements, application logs, clickstreams and security alerts.

5.1 Event brokers and logs

Platforms such as Apache Kafka organise events into topics and partitions. Producers append records, while consumers read them, often tracking offsets. Partitioning enables parallel consumption, and retention allows consumers to replay data. Ordering is usually guaranteed within a partition rather than across an entire topic.

A robust event record commonly includes an event identifier, event time, producer information, schema version, payload and sometimes a correlation key. Schemas and compatibility rules are essential because producers and consumers evolve independently.

5.2 Processing time, event time and watermarks

Processing time is when a system handles an event. Event time is when the event actually occurred. Networks, devices and temporary failures can cause late or out-of-order events, so event-time windows are generally more meaningful for business analysis.

A tumbling window divides time into non-overlapping intervals, such as each five-minute period. A sliding window has a window size and a slide interval, so windows may overlap. A session window groups events separated by less than a specified inactivity gap.

A watermark is an engine’s estimate that events earlier than a particular event-time threshold are unlikely to arrive. Watermarks allow a system to emit results while defining a policy for late data. They do not prove that no later event will arrive. Applications must choose whether to drop, update, retract or separately reconcile late events.

5.3 Delivery and processing guarantees

At-most-once processing may lose events but avoids duplicates. At-least-once processing retries failures but can produce duplicates. Exactly-once semantics aim to make each event’s effect appear once within a defined system boundary; they require coordinated state, checkpoints, transactional sinks or idempotent operations. Exactly-once is not a universal property of an entire business process.

An idempotent operation produces the same final result when safely repeated. For example, setting an account status to “verified” is more naturally idempotent than adding a payment amount repeatedly. Event identifiers, deduplication keys and transactional design help manage retries.

5.4 Lambda and Kappa perspectives

A Lambda architecture uses separate batch and speed layers, combining comprehensive historical recomputation with low-latency results. It can provide resilience but duplicates logic and increases operational complexity. A Kappa-style architecture uses a streaming pipeline and reprocesses retained events when necessary. It simplifies conceptual flow but requires durable event retention and a capable stream processor. Modern platforms often implement a pragmatic hybrid rather than following either model rigidly.

6. Cloud Data Warehouses, Data Lakes and Lakehouses

6.1 Cloud data warehouses

Cloud warehouses provide managed, SQL-oriented analytical storage. They commonly separate compute from storage, use columnar formats, compress data and execute queries in parallel. Independent compute scaling supports workloads ranging from dashboards to large transformations.

Warehouses are strong in governed structured analytics, concurrency management, SQL performance and integration with business intelligence. Costs require attention because charges may depend on storage, compute time, scanned data or reserved capacity. Poorly bounded queries and repeated transformations can create significant expenditure.

6.2 Data lakes

A data lake stores raw or lightly processed data in inexpensive object storage. It can hold structured, semi-structured and unstructured data, including images, logs, documents and machine-learning features. Open file formats such as Parquet support columnar analytical access and interoperability.

A lake is not automatically useful merely because it contains large quantities of data. Without catalogues, ownership, quality controls, access policies and lifecycle management, it can become a “data swamp.” Zones such as raw, refined and curated are organisational conventions that help establish processing responsibility.

6.3 Lakehouses

A lakehouse combines the flexible storage and open formats of a lake with warehouse-like table management, transactions, schema enforcement and reliable analytical performance. Table formats such as Delta Lake, Apache Iceberg and Apache Hudi provide capabilities including snapshots, schema evolution and time travel, although exact features and implementations differ.

Lakehouses are valuable where one copy of data should support BI, data science, machine learning and streaming. They still require careful governance, optimisation, cataloguing and workload isolation. A lakehouse is an architectural pattern, not a guarantee of low cost or high quality.

7. Modern Data-Stack Architectures

A contemporary data stack often includes:

1. Sources: operational databases, SaaS applications, devices and event producers.

2. Ingestion: batch extraction, change data capture, APIs and event brokers.

3. Storage: object stores, warehouses, lakehouses and operational databases.

4. Transformation: SQL transformation tools, Spark, stream processors and data-quality jobs.

5. Orchestration: scheduling, dependency management, retries and monitoring.

6. Serving: dashboards, semantic layers, APIs, feature stores and reverse-ETL destinations.

7. Governance: catalogues, lineage, identity, policy, quality and retention controls.

The older ETL pattern transforms data before loading it into a warehouse. ELT loads data first and performs transformations in the analytical platform. ELT is attractive when cloud compute is elastic and raw data must be preserved, but it does not remove the need for validation and controlled access.

Data mesh is an organisational and architectural approach that treats domain teams as owners of governed data products, while a central platform team supplies self-service infrastructure and standards. It is not simply a technology product. Its success depends on clearly defined ownership, interoperability and accountability.

A semantic layer defines consistent business meanings for measures such as revenue, active customer and churn. Without it, different dashboards may calculate the same term differently, undermining trust in analytics.

8. Selecting a Platform: A Decision Framework

Platform selection should begin with workload requirements rather than product names. Key questions include:

• What is the data volume and expected growth?
• What latency is required: seconds, minutes, hours or days?
• Are queries predictable or exploratory?
• Is the data structured, semi-structured, graph-shaped or unstructured?
• What consistency and transaction guarantees are required?
• How much data must be replayed or retained?
• What skills, operating model and budget are available?
• What privacy, residency, audit and deletion requirements apply?

A relational warehouse is often appropriate for governed reporting and dimensional analysis. An object-storage lake or lakehouse is attractive for diverse data and machine-learning workflows. A NoSQL operational store is appropriate for high-volume application access with known query patterns. A streaming platform is necessary when decisions depend on events shortly after they occur.

A mature architecture may use several technologies. Polyglot persistence is justified when each store has a clear role, but it increases data movement, operational complexity and governance burden. The number of systems should therefore be minimised unless the benefits are demonstrable.

9. Worked Design Example: Real-Time Fraud Monitoring

Consider an illustrative online-payment company processing card transactions, customer profiles and merchant information. It wants to detect suspicious payments within seconds, provide daily regulatory reports and train models using several years of history.

Background and problem

Transactions arrive continuously, some devices submit events late, and duplicate messages may occur after network retries. Analysts need SQL access to historical data, while an operational service must return a risk score rapidly. Personal data must be protected and decisions must be auditable.

Analysis

A single nightly warehouse load cannot meet the real-time requirement. A graph database may assist with relationship investigation, but it is not by itself a complete historical analytics platform. The design must support event ingestion, stateful features, durable history, low-latency scoring and reproducible audit records.

Approach

1. Transaction events are published to partitioned topics using a transaction or account key where suitable.

2. A stream processor validates schemas, removes duplicates using event identifiers and computes event-time features such as transaction frequency within a defined window.

3. A low-latency feature or key-value store serves current features to the scoring service.

4. All accepted events are retained in object storage using an open columnar format and governed table metadata.

5. A lakehouse or cloud warehouse supports daily reporting, investigation and model-training datasets.

6. Failed events are placed in a quarantine stream with monitoring and replay procedures.

7. Access policies restrict raw personal data; tokenisation or pseudonymisation is used where direct identity is unnecessary.

8. Model versions, input features, scores, analyst decisions and final outcomes are recorded for audit and model evaluation.

Outcome

The architecture supports rapid decisions and complete historical recomputation. It does not eliminate trade-offs: stream processing requires careful state management, late events can change results, and maintaining both serving and analytical stores creates synchronisation responsibilities.

Learning points

The case illustrates that latency, durability, analytical flexibility and governance are separate requirements. It also demonstrates why event identifiers, schema evolution, replayability, data lineage and clearly defined ownership are as important as raw processing speed.

10. Governance, Security and Responsible Use

Scalable platforms increase the number of copies, users and processing paths through which data can move. Governance should therefore be designed into the platform rather than added after deployment.

Important controls include encryption in transit and at rest, identity-based access, least privilege, network isolation, secrets management, audit logs, retention schedules, deletion workflows and classification of sensitive fields. Privacy principles such as purpose limitation and data minimisation are relevant when collecting and reusing personal data. Applicable laws may include data-protection, financial-sector, health or sector-specific requirements; organisations must obtain authoritative legal advice for their jurisdiction.

Data quality dimensions include accuracy, completeness, consistency, timeliness, uniqueness and validity. Quality checks should be observable and actionable. A pipeline that silently produces incorrect data can be more dangerous than one that fails visibly.

Machine-learning systems add concerns about bias, explainability, drift and feedback loops. Real-time decisions may affect access to finance, employment, safety or essential services. Teams should document model purpose, training data, decision thresholds, monitoring procedures and human escalation paths.

11. Challenges and Contemporary Developments

Common technical challenges include data skew, small files, schema drift, duplicate events, hot partitions, inefficient joins, cloud egress charges and unbounded state. Reliability requires checkpoints, retries, back-pressure, dead-letter handling, capacity planning and disaster recovery tests.

Contemporary developments include serverless and managed analytics, open table formats, vectorised query engines, real-time lakehouse ingestion, data observability, streaming feature stores and AI-assisted data engineering. Generative AI can help write queries or documentation, but generated code and transformations require testing, access control and human review. It should not be granted unrestricted access to sensitive data.

Best practices include designing from access patterns, using open formats where interoperability matters, partitioning carefully, measuring end-to-end latency, testing failure scenarios, versioning schemas, monitoring cost and quality, and documenting ownership. Benchmarking should use representative data and realistic concurrency rather than a small demonstration dataset.

Chapter Summary

Scalable data platforms distribute storage and computation to handle large, fast and diverse workloads. Horizontal scaling improves capacity but introduces partitioning, coordination and consistency concerns. Hadoop established durable distributed storage and batch processing through HDFS and MapReduce, while Spark offers DAG-based execution for batch, SQL, streaming and machine-learning workloads. NoSQL systems provide specialised key-value, document, wide-column and graph models and should be selected according to access patterns.

Streaming platforms process unbounded event flows using partitions, offsets, windows, watermarks and delivery guarantees. Cloud warehouses provide governed SQL analytics; data lakes provide flexible, economical storage; and lakehouses seek to combine both advantages through managed open tables. Modern data stacks integrate ingestion, storage, transformation, orchestration, serving and governance. Effective platform design balances latency, scalability, reliability, cost, interoperability, security and responsible use rather than pursuing technology novelty alone.

Key Terms

Distributed storage: Storage that partitions and often replicates data across multiple nodes.

Horizontal scaling: Increasing capacity by adding machines or service instances.

Replication: Maintaining multiple copies of data for durability or availability.

Partitioning: Dividing data or work into independently managed segments.

CAP theorem: A model describing the consistency–availability trade-off during network partition.

HDFS: Hadoop’s distributed file system for large-scale, fault-tolerant storage.

MapReduce: A distributed programming model based on map, shuffle and reduce stages.

Apache Spark: A distributed engine for batch, SQL, streaming and machine-learning workloads.

Shuffle: Network redistribution of intermediate data between processing tasks.

NoSQL: A family of non-relational databases designed for specialised scalability or data models.

Event time: The time at which an event occurred in the source domain.

Watermark: An estimate of how far event-time processing has progressed.

At-least-once processing: Processing that may retry events and therefore produce duplicates.

Data warehouse: Managed analytical storage optimised for structured queries and reporting.

Data lake: Flexible storage, commonly in object storage, for diverse raw and processed data.

Lakehouse: An architecture combining lake storage with warehouse-like table management.

Change data capture: Recording inserts, updates and deletes from a source system.

Data lineage: Documentation of data origins, transformations and destinations.

Semantic layer: A governed definition of business entities, measures and dimensions.

Data mesh: A domain-oriented approach to data ownership supported by a shared platform.

Review Questions

Short Answer Questions

1. Distinguish vertical scaling from horizontal scaling.

2. What is the purpose of replication in distributed storage?

3. Why are Spark shuffles often expensive?

4. Differentiate processing time from event time.

5. What distinguishes a data lake from a cloud data warehouse?

Long Answer Questions

1. Compare HDFS and MapReduce with Spark, including their strengths and limitations.

2. Explain the four major NoSQL data models and provide a suitable application for each.

3. Discuss windows, watermarks and late events in stream processing.

4. Compare data warehouses, data lakes and lakehouses in terms of storage, governance and workloads.

5. Explain the major components and responsibilities of a modern data-stack architecture.

Analytical / Application Questions

1. Design a platform for a smart-city system that receives sensor events, supports real-time alerts and produces monthly planning reports. Justify storage, processing and governance choices.

2. A Spark job has 20% sequential work and 16 parallel workers. Using Amdahl’s law, calculate its ideal speedup and explain why actual speedup may be lower.

3. A retailer experiences duplicate and late order events in a real-time revenue dashboard. Propose an event-processing and reconciliation strategy that addresses idempotency, event time, watermarks and auditability.

Multiple Choice Questions with Answers

Question 1

Which characteristic is most closely associated with horizontal scaling?

A. Adding CPU capacity to one server

B. Adding servers and distributing work among them

C. Replacing structured data with images

D. Removing replication from storage

Correct Answer: B. Adding servers and distributing work among them

Question 2

What does a Spark shuffle generally involve?

A. Only local computation within one partition

B. Network redistribution of intermediate data

C. Deleting unused source records

D. Encrypting every output file

Correct Answer: B. Network redistribution of intermediate data

Question 3

Which NoSQL model is most naturally suited to relationship-intensive queries such as fraud networks?

A. Key-value

B. Document

C. Graph

D. Wide-column

Correct Answer: C. Graph

Question 4

What is event time?

A. The time when an event actually occurred in its source domain

B. The time when a query is submitted

C. The time when a database backup starts

D. The time when a user logs out

Correct Answer: A. The time when an event actually occurred in its source domain

Question 5

What is the primary purpose of a watermark in stream processing?

A. To encrypt late events

B. To estimate progress in event-time processing

C. To increase the number of partitions automatically

D. To replace schema validation

Correct Answer: B. To estimate progress in event-time processing

Question 6

Which platform is generally most associated with governed SQL analytics and business intelligence?

A. Cloud data warehouse

B. Session cache

C. Graph traversal engine only

D. Message producer

Correct Answer: A. Cloud data warehouse

Question 7

Why can a data lake become a data swamp?

A. It contains too many indexes

B. It lacks cataloguing, ownership, quality and governance

C. It uses only relational tables

D. It cannot store unstructured data

Correct Answer: B. It lacks cataloguing, ownership, quality and governance

Question 8

What does at-least-once processing imply?

A. Events can never be retried

B. Events may be processed more than once

C. Events are always lost

D. Processing must be entirely sequential

Correct Answer: B. Events may be processed more than once

Question 9

If 10% of a workload is sequential and 8 workers process the rest, what is the ideal Amdahl speedup?

A. 2.50

B. 4.71

C. 8.00

D. 10.00

Correct Answer: B. 4.71

Question 10

Which practice best supports reliable evolution of event-driven systems?

A. Changing schemas without notifying consumers

B. Removing event identifiers

C. Using versioned schemas and compatibility rules

D. Disabling replay and retention

Correct Answer: C. Using versioned schemas and compatibility rules