8. Big Data and Distributed Computing Platforms

Introduction and Context

Data science increasingly operates on datasets whose volume, speed, and diversity exceed the capabilities of a single computer. Transaction records, application logs, sensor streams, social-media events, scientific measurements, images, and machine-generated telemetry may arrive continuously and must often be stored and analysed at scale. The central problem is therefore not merely how to write an algorithm, but how to organise computation and storage across many machines while preserving acceptable performance, reliability, security, and cost.

Distributed computing addresses this problem by dividing data and computation among networked computers. A distributed platform coordinates these machines so that, from the user’s perspective, they operate as a logical analytical system. The approach offers horizontal scalability: capacity is increased by adding machines rather than replacing one server with a substantially larger server. However, distribution also introduces complexity. Networks can fail, machines can be slow or unavailable, data may be duplicated, and different tasks may compete for resources.

This chapter examines the principal platforms and design ideas used in big-data analytics. It covers Hadoop and its ecosystem, Spark’s unified processing model, streaming systems, distributed query engines, data-lake and lakehouse architectures, and the practical decisions involved in choosing among them. The emphasis is on understanding how platforms work, when they are appropriate, and how technical choices affect organisational outcomes.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the characteristics of big data and the motivation for distributed storage and processing.

2. Describe the architecture and operating principles of Hadoop, HDFS, YARN, MapReduce, and related ecosystem tools.

3. Explain Spark’s execution model, abstractions, and suitability for batch, interactive, machine-learning, and streaming workloads.

4. Distinguish batch processing, micro-batch processing, and event-at-a-time stream processing.

5. Compare distributed query engines and storage architectures, including data warehouses, data lakes, and lakehouses.

6. Select an appropriate platform by considering latency, throughput, reliability, governance, skills, and cost.

7. Identify security, privacy, operational, and data-quality risks in distributed environments.

8. Design a high-level scalable analytics architecture for a realistic organisational problem.

Why Distribution Is Necessary

Big data is commonly described through dimensions such as volume, velocity, variety, veracity, and value. These dimensions are useful, but they do not by themselves define a precise technical threshold. A dataset becomes a big-data problem when its size, arrival rate, complexity, or processing requirements make conventional single-node methods inadequate or economically unattractive.

A single machine has finite storage capacity, memory, network bandwidth, and processing power. Even when a large server can hold a dataset, processing it sequentially may take too long for operational or analytical purposes. Distributed systems divide a dataset into partitions and execute tasks in parallel. If a file is divided into ten partitions and ten workers process them concurrently, elapsed time may decrease substantially, although the improvement is limited by coordination, data movement, skew, and the slowest task.

Two ideas are especially important. Data partitioning determines how records are distributed across machines. Data locality attempts to execute computation near the data, reducing network transfer. In distributed analytics, moving computation is often cheaper than moving very large datasets across the network.

Distribution does not eliminate failure; it assumes failure will occur. A worker may crash during a computation, a disk may become unavailable, or a network connection may be interrupted. Platforms therefore use replication, checkpointing, task retry, lineage, or recomputation. Reliability is achieved through coordinated mechanisms rather than dependence on a single highly reliable machine.

A distributed system must also manage the CAP trade-off: in the presence of a network partition, a system cannot simultaneously guarantee perfect consistency and complete availability. Different big-data platforms make different choices depending on whether the workload prioritises immediate consistency, high availability, or eventual convergence. CAP should not be treated as a complete description of system quality, but it provides a useful lens for analysing distributed data services.

Hadoop: The Foundational Distributed Data Platform

Hadoop established an influential open-source model for storing and processing very large datasets on clusters of commodity servers. Its principal components are the Hadoop Distributed File System (HDFS), YARN, MapReduce, and a broad ecosystem of related tools.

HDFS and Distributed Storage

HDFS stores large files by dividing them into blocks and distributing those blocks across DataNodes. A NameNode maintains metadata about files, directories, permissions, and block locations, while DataNodes store the actual blocks. Blocks are replicated so that loss of one machine does not necessarily result in data loss. The replication factor is a configuration choice that affects resilience, storage overhead, and performance.

HDFS is optimised for large sequential reads and writes. It is therefore well suited to analytical files such as logs, historical transactions, and large extracts. It is not designed as a low-latency database for millions of small updates. The small-files problem illustrates this limitation: a very large number of small files creates excessive metadata and scheduling overhead, even when their combined size is modest.

Modern deployments have reduced dependence on local HDFS through object storage such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage. Object stores provide durable, scalable storage, while processing engines run separately and access the stored data over network interfaces. This separation of storage and compute improves elasticity but makes file formats, partitioning, caching, and network costs especially important.

YARN and Cluster Resource Management

YARN separates resource management from data processing. A ResourceManager allocates cluster resources, while NodeManagers supervise resources on individual machines. Applications request containers in which tasks execute. This architecture allows several processing frameworks to share a cluster rather than dedicating the entire environment to MapReduce.

Resource management involves more than assigning CPU and memory. Production clusters must account for queues, priorities, workload isolation, service-level objectives, fair scheduling, and failure recovery. Poor resource governance can allow one large job to delay interactive queries or streaming applications.

MapReduce

MapReduce expresses computation through two conceptual phases. The map phase transforms input records into intermediate key-value pairs. The shuffle and sort phase groups values by key, and the reduce phase aggregates or processes each group. For example, a word-count job maps each word to the pair (word, 1), then reduces all values associated with a word by summing them.

MapReduce is powerful because it provides automatic parallelisation, task retry, and fault tolerance. Its limitation is that intermediate results are commonly written to disk between stages. Algorithms involving repeated iterations, interactive exploration, or complex multi-stage pipelines may therefore experience high latency. MapReduce remains historically important and can still be appropriate for simple, very large batch jobs, but many organisations now use Spark or cloud-native engines for general analytics.

Spark’s Unified Processing Model

Apache Spark was designed to provide faster and more flexible distributed computation than traditional disk-oriented MapReduce. Spark can run on cluster managers such as YARN, Kubernetes, or its own standalone manager, and can read from HDFS, object stores, relational databases, and streaming systems.

Spark applications contain a driver program and executor processes. The driver constructs a logical plan and coordinates execution. Executors run tasks, hold cached data, and return results. Spark uses a directed acyclic graph (DAG) of transformations. Operations are divided into stages at points where data must be redistributed, commonly called shuffle boundaries.

Spark supports low-level RDDs, DataFrames, and SQL. An RDD is a fault-tolerant distributed collection whose partitions can be recomputed from lineage. DataFrames provide named columns and enable the engine to optimise queries. Spark SQL uses a catalyst optimiser and a physical execution engine to improve plans, push filters, select efficient joins, and reduce unnecessary data movement. In professional work, DataFrames and SQL are generally preferred unless specialised RDD behaviour is required.

Spark transformations are usually lazy: they describe a computation without executing it immediately. An action, such as writing output or counting records, triggers execution. Laziness allows the engine to optimise the complete plan, but it can also surprise beginners who expect each line of code to produce immediate results.

Caching can improve iterative workloads by retaining reusable partitions in memory or on disk. It should not be applied indiscriminately. Caching data that is used only once wastes memory and may cause eviction or spilling. Effective Spark design includes selecting appropriate partition counts, avoiding unnecessary shuffles, filtering early, choosing suitable join strategies, and monitoring execution plans.

Spark Structured Streaming extends the DataFrame and SQL model to continuously arriving data. A stream is processed using a query with defined sources, transformations, and sinks. Depending on configuration and workload, execution may use micro-batches or continuous processing approaches. Checkpointing records progress and supports recovery. Exactly-once behaviour is not a universal property of every source and sink; it depends on the complete processing design, including replayability and idempotent output.

Batch, Streaming, and Event-Time Analytics

Batch processing collects data and processes it in scheduled or manually triggered jobs. It is suitable for payroll, daily financial reporting, historical model training, and large transformations where immediate results are unnecessary. Batch workloads normally optimise throughput and cost rather than millisecond latency.

Streaming processing analyses data as it arrives. Typical use cases include fraud detection, equipment monitoring, dynamic pricing, clickstream analysis, and operational alerting. Streaming systems must address out-of-order events, duplicate messages, late arrivals, changing schemas, state management, and recovery.

A central distinction is between processing time, when a platform receives or processes an event, and event time, when the event actually occurred. Event-time windows produce more meaningful results when network delay or temporary disconnection causes events to arrive late. A watermark is an operational assumption about how late events may be before the system finalises a result. A system that uses a five-minute watermark may revise or reject events that arrive beyond that threshold, depending on its configuration.

Kafka is a widely used distributed event-streaming platform. Producers write records to topics, which are divided into partitions. Consumers read records, often as members of a consumer group. Partitioning provides parallelism, and offsets allow consumers to track progress and replay messages. Kafka is not simply a database or a message queue; it is a durable, ordered log within each partition. Ordering across all partitions is not automatically guaranteed.

Other streaming systems, including Apache Flink and cloud-managed event services, provide event-time processing, stateful operators, windows, and fault-tolerant checkpoints. The choice depends on latency requirements, operational skills, ecosystem integration, state complexity, and cloud strategy.

Distributed Query Engines and Data Architectures

A distributed query engine allows analysts to query data stored across multiple nodes or services. Trino, Presto, Impala, BigQuery, Snowflake, Redshift, and Synapse represent different approaches to distributed analytical SQL. Some are engines that query external data sources; others combine storage, compute, metadata, and administration in a managed warehouse.

Query engines divide a SQL statement into stages and tasks. A coordinator creates the plan, while workers scan files, filter rows, perform joins, aggregate results, and exchange intermediate data. Performance depends heavily on predicate pushdown, column pruning, statistics, partition elimination, join distribution, file size, and data format.

Row-oriented formats store complete records together and are often useful for transactional access. Columnar formats such as Parquet and ORC store values from the same column together. For analytical queries that read only some columns, columnar storage reduces I/O and improves compression. Compression is usually beneficial because the cost of reading fewer bytes can exceed the cost of decompression.

A data warehouse typically stores curated, structured data with strong schema management and high-performance reporting. A data lake stores raw and processed data in flexible formats, often at lower storage cost, but requires disciplined governance to avoid becoming a disorganised data swamp. A lakehouse attempts to combine the openness and scale of a lake with warehouse-like transactions, schema enforcement, cataloguing, and reliable table management. Technologies such as Delta Lake, Apache Iceberg, and Apache Hudi support table formats with features including transaction logs, schema evolution, time travel, and incremental processing.

The separation of storage and compute allows organisations to scale processing resources independently from storage. However, it introduces architectural responsibilities: data must be partitioned sensibly, lifecycle policies must control obsolete files, access must be governed consistently, and query engines must be selected according to workload rather than brand familiarity.

A Framework for Platform Selection

Platform selection should begin with workload characteristics rather than with a preferred technology. The following questions provide a practical decision framework:

• How much data exists now, and how quickly will it grow?
• Is the workload batch, interactive, operational streaming, or a combination?
• What latency is required: seconds, minutes, hours, or days?
• Are queries predictable and structured, or exploratory and diverse?
• Does the workload require joins, iterative machine learning, stateful processing, or graph analysis?
• What consistency, replay, recovery, and audit requirements apply?
• Which data sources, cloud services, programming languages, and skills already exist?
• What are the total costs of storage, compute, data transfer, administration, and failure?

A small dataset requiring low-latency transactional updates should not automatically be placed in Hadoop or Spark. Conversely, a large historical dataset requiring repeated transformations may be unsuitable for a single relational server. A mature architecture may use several platforms: Kafka for ingestion, object storage for durable raw data, Spark for transformation and model training, and a warehouse or query engine for governed consumption.

Applied Case: Retail Demand and Inventory Intelligence

Background

Consider an illustrative retailer, Northstar Retail, operating 300 stores and an online channel. The organisation collects point-of-sale transactions, web events, inventory updates, supplier deliveries, and temperature data from refrigerated storage. Existing reports are produced overnight from operational databases and cannot reliably identify stock-out risk during the trading day.

Situation and Problem

Northstar wants store-level demand forecasts updated hourly and alerts when a product is likely to become unavailable. Data arrives at different speeds and with different quality. Online events may arrive within seconds, while supplier files may arrive once per day. Product identifiers are inconsistent across systems, and network interruptions sometimes produce duplicate events.

Analysis

The problem has both streaming and batch dimensions. Current inventory changes and sales events require low-latency ingestion and processing. Historical transactions are needed for forecasting and can be processed in scheduled jobs. The organisation also needs replay because a faulty transformation must not permanently corrupt the analytical record.

Approach and Solution

Northstar introduces an event-streaming layer in which sales and inventory events are published to partitioned topics. A schema registry and validation process reduce incompatible changes. Events are stored in durable object storage in a raw zone, preserving the original payload and ingestion metadata.

A streaming application performs deduplication using event identifiers, standardises product codes, and maintains event-time windows. Watermarks accommodate delayed store connectivity. The application writes curated hourly aggregates to a governed lakehouse table and sends urgent alerts to an operations service.

A nightly Spark job processes historical data, supplier files, promotions, weather data, and corrected records. It creates training features and generates demand forecasts. A distributed SQL engine provides analysts with interactive access to curated tables, while dashboards display inventory risk by store and product.

Security controls restrict personally identifiable information. Role-based access separates store managers, analysts, data engineers, and data scientists. Data-quality checks measure missing identifiers, duplicate rates, late-event percentages, and reconciliation against financial totals.

Outcome

The architecture supports both immediate operational decisions and historical analysis. When a transformation error occurs, raw events can be replayed. Separating raw and curated layers improves traceability, while shared table formats reduce unnecessary copying between processing and query systems. The solution does not remove all problems: it requires monitoring, platform expertise, schema governance, and careful cost management.

Learning Points

The case demonstrates that a single platform rarely solves every analytical requirement. Streaming and batch processing should share definitions and governance but may use different execution paths. Durable raw data, replayability, data-quality measurement, and explicit latency requirements are architectural assets. Most importantly, technology should be selected in relation to business decisions, not merely data volume.

Operational Risks, Governance, and Professional Practice

Distributed platforms create technical and organisational risks. Data skew occurs when a small number of partitions contain disproportionately large amounts of data, causing a few tasks to run much longer than the rest. Small files increase metadata and scheduling overhead. Excessive shuffles consume network and disk resources. Poorly designed joins may cause out-of-memory failures or long-running tasks.

Streaming systems introduce additional risks. Duplicate delivery, lost offsets, late events, non-idempotent sinks, and incorrect window logic can produce misleading results. A dashboard that appears current may nevertheless contain incomplete data if freshness and completeness are not monitored separately.

Security must be designed across storage, compute, network, identities, and metadata. Important controls include encryption in transit and at rest, strong authentication, least-privilege authorisation, secret management, audit logs, network segmentation, and controlled administrative access. Sensitive attributes may require masking, tokenisation, aggregation, or removal. Data retention should reflect legal, contractual, and business requirements rather than unlimited preservation.

Governance concerns include data ownership, lineage, cataloguing, quality standards, schema evolution, access review, and reproducibility. Regulations concerning personal data may grant rights related to access, correction, deletion, purpose limitation, and cross-border transfer. Distributed copies and derived features make deletion and correction technically difficult; governance must therefore track where data is replicated and transformed.

Professional teams should monitor throughput, latency, error rates, consumer lag, task failures, storage growth, query cost, and data-quality indicators. Observability should connect infrastructure metrics with business measures. For example, a technically healthy pipeline may still be failing if the number of processed transactions is far below expected levels.

Emerging Directions

Several developments are shaping distributed analytics. Cloud-managed services reduce infrastructure administration and allow elastic computation, but they can create vendor dependence and unpredictable expenditure. Serverless query engines make exploratory analysis accessible, although poorly designed queries may generate high costs.

Lakehouse table formats are becoming important because they provide transactions, schema management, and incremental processing over scalable object storage. Vectorised execution, adaptive query optimisation, and specialised hardware improve analytical performance. Streaming and batch systems are increasingly converging around unified APIs and shared storage.

The growth of machine learning introduces feature stores, distributed model training, model monitoring, and retrieval systems. Generative AI applications also increase demand for scalable document processing, embedding pipelines, vector search, and governance over training and retrieval data. These applications do not replace conventional data engineering; they depend on reliable ingestion, metadata, access control, and quality assurance.

Sustainability is another consideration. Distributed systems consume energy through storage replication, data movement, idle clusters, and repeated computation. Efficient formats, autoscaling, workload scheduling, caching, and lifecycle policies can reduce both financial and environmental costs.

Best Practices for Implementation

A reliable distributed analytics programme generally follows these principles:

1. Begin with measurable business and service requirements, including latency, freshness, availability, and cost.

2. Use durable, open, and appropriately partitioned storage formats.

3. Design for failure, replay, idempotency, and recovery from the beginning.

4. Prefer columnar formats and avoid unnecessary data movement.

5. Control file sizes, partition counts, and schema evolution.

6. Separate raw, validated, and curated data while maintaining lineage.

7. Automate testing for schema, completeness, uniqueness, validity, and reconciliation.

8. Apply security and privacy controls before production deployment.

9. Monitor both technical performance and business outcomes.

10. Treat platform operations, documentation, and cost management as part of data science rather than as secondary administration.

Chapter Summary

Distributed computing enables large-scale analytics by partitioning data and computation across multiple machines. Hadoop established the durable-storage and cluster-processing model through HDFS, YARN, and MapReduce. Spark provides a more flexible execution engine based on DAGs, lazy transformations, DataFrames, SQL, caching, and structured streaming. Batch processing prioritises high-throughput computation, whereas streaming systems address continuous events, state, windows, late data, and replay.

Distributed query engines rely on parallel scans, columnar storage, predicate pushdown, partition pruning, and efficient joins. Data warehouses, data lakes, and lakehouses represent different combinations of governance, flexibility, performance, and cost. Effective platform selection depends on workload, latency, reliability, governance, skills, and total cost rather than data volume alone. Professional implementation requires attention to failure, security, privacy, data quality, observability, and responsible use of resources.

Key Terms

Big data: Data whose volume, velocity, variety, or processing requirements exceed convenient single-machine handling.

Distributed computing: Execution of storage or computation across multiple networked machines.

Horizontal scalability: Increasing capacity by adding machines to a system.

Data locality: Executing computation near the data to reduce network transfer.

Partition: A logical division of data processed or stored independently.

Replication: Maintaining multiple copies of data to improve fault tolerance or availability.

HDFS: Hadoop’s distributed file system for storing large files across cluster nodes.

NameNode: HDFS service that maintains filesystem metadata and block locations.

YARN: Hadoop resource-management and application-scheduling framework.

MapReduce: A distributed programming model based on map, shuffle, and reduce stages.

RDD: Spark’s fault-tolerant distributed collection reconstructed through lineage.

DataFrame: Spark’s structured distributed dataset with optimisable columns and rows.

DAG: Directed acyclic graph representing dependencies among computation stages.

Shuffle: Redistribution of data across partitions, commonly for joins or aggregations.

Event time: The time at which an event actually occurred.

Watermark: A progress estimate used to manage late-arriving streaming events.

Data lake: Scalable storage containing raw and processed data in varied formats.

Data warehouse: Governed analytical storage designed primarily for structured querying and reporting.

Lakehouse: Architecture combining lake-scale storage with warehouse-like table management and governance.

Idempotency: Property whereby repeating an operation produces the same effective result as performing it once.

Review Questions

Short Answer Questions

1. Why is data locality valuable in distributed processing?

2. What roles do the NameNode and DataNodes perform in HDFS?

3. Distinguish a Spark transformation from a Spark action.

4. What is the difference between processing time and event time?

5. Why are columnar formats useful for analytical queries?

Long Answer Questions

1. Explain the architecture of Hadoop and discuss the strengths and limitations of HDFS and MapReduce.

2. Describe Spark’s execution model, including the driver, executors, DAG, stages, lazy evaluation, and caching.

3. Compare batch processing with streaming processing and explain the importance of windows, watermarks, and checkpointing.

4. Compare data warehouses, data lakes, and lakehouses in terms of flexibility, governance, performance, and operational requirements.

5. Develop a framework for selecting a distributed analytics platform for a large organisation.

Analytical / Application Questions

1. A company receives millions of sensor events per hour but needs only a daily compliance report. Would you recommend a streaming architecture, a batch architecture, or a combination? Justify your choice.

2. A Spark job has one task running for an hour while all other tasks finish in minutes. Analyse the likely cause and propose corrective actions.

3. Design a high-level architecture for fraud detection that must process transactions within seconds while also supporting historical model training and regulatory audit.

22. Multiple Choice Questions with Answers

Question 1

A distributed analytics job experiences very long network transfers because computation is performed far from the stored data. Which principle has been neglected?

A. Data locality

B. Schema evolution

C. Column pruning

D. Event-time watermarking

Correct Answer: A. Data locality

Question 2

Which Hadoop component is primarily responsible for managing filesystem metadata and block locations?

A. DataNode

B. NameNode

C. NodeManager

D. Reducer

Correct Answer: B. NameNode

Question 3

Why can Spark outperform traditional MapReduce for iterative machine-learning workloads?

A. It eliminates all network communication

B. It can cache reusable data and avoid repeated disk materialisation

C. It requires only one worker machine

D. It does not perform any data partitioning

Correct Answer: B. It can cache reusable data and avoid repeated disk materialisation

Question 4

A streaming application must group events according to when they actually occurred, even when some arrive late. Which concepts are most relevant?

A. Processing time and replication factor

B. Event time and watermarks

C. Map and reduce functions

D. Object storage and compression

Correct Answer: B. Event time and watermarks

Question 5

What is a likely consequence of severe data skew in a distributed job?

A. All partitions finish simultaneously

B. One or a few tasks process disproportionately large partitions

C. The system automatically gains more memory

D. The data becomes strongly consistent

Correct Answer: B. One or a few tasks process disproportionately large partitions

Question 6

Which feature most directly supports replay after a streaming transformation error?

A. Durable event retention and tracked offsets

B. Deleting raw events after processing

C. Using only in-memory storage

D. Removing event identifiers

Correct Answer: A. Durable event retention and tracked offsets

Question 7

Why do columnar formats such as Parquet commonly improve analytical-query performance?

A. They force every query to read every column

B. They enable selective column reads and effective compression

C. They prevent data partitioning

D. They are designed primarily for single-row transactional updates

Correct Answer: B. They enable selective column reads and effective compression

Question 8

An organisation wants open object storage with transaction support, schema management, and reliable incremental updates. Which architecture is most appropriate?

A. A lakehouse

B. A raw message queue only

C. A single-node spreadsheet system

D. A traditional file server without metadata

Correct Answer: A. A lakehouse

Question 9

Which design best supports safe repeated writes from a streaming pipeline?

A. Non-idempotent updates with no event identifiers

B. Idempotent processing using stable event identifiers

C. Deleting checkpoints after every batch

D. Ignoring duplicate delivery

Correct Answer: B. Idempotent processing using stable event identifiers

Question 10

When selecting between a warehouse, Spark, and a streaming engine, the most appropriate first step is to:

A. Choose the most popular technology

B. Assess workload requirements such as latency, volume, consistency, skills, and cost

C. Store all data permanently in memory

D. Avoid defining service-level objectives

Correct Answer: B. Assess workload requirements such as latency, volume, consistency, skills, and cost