Big Data Ecosystems and Architecture

Introduction

Big data is not simply a large database. It is an organisational and technical capability for collecting, storing, processing, governing and analysing data whose volume, velocity, variety, variability and value exceed the practical limits of conventional systems. A modern analytics solution may combine operational databases, application programming interfaces (APIs), event streams, cloud object storage, distributed processing engines, machine-learning platforms and business-intelligence tools.

The central architectural question is not “Which technology is best?” It is “Which combination of technologies provides the required performance, reliability, governance, cost and analytical flexibility?” A transaction-processing system must protect the correctness of individual updates, whereas an analytical platform may prioritise large-scale scans, historical retention and rapid experimentation. Confusing these purposes produces expensive and fragile systems.

This chapter compares the principal components of big-data ecosystems: databases, data warehouses, data lakes, lakehouses, cloud platforms, distributed storage, batch and streaming architectures, Hadoop, Spark and NoSQL technologies. It also examines governance, security, architecture selection and the implications for data-driven strategy.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish operational databases, data warehouses, data lakes and lakehouses.
2. Explain distributed storage and processing using Hadoop and Spark.
3. Compare relational and NoSQL database models.
4. Differentiate batch, micro-batch and event-streaming architectures.
5. Evaluate cloud deployment choices and architecture trade-offs.
6. Design a suitable architecture for a business analytics problem.
7. Identify governance, ethical, security and regulatory requirements.

1. The Big-Data Ecosystem

A big-data ecosystem is a set of interoperating technologies and organisational practices. Its layers commonly include:

An architecture is effective when these layers work as a coherent system. For example, a fraud-detection service may ingest card events within seconds, enrich them with customer and merchant data, apply a model, record the decision and send an alert. A financial-reporting platform may instead load validated records nightly and calculate reconciled monthly measures.

The distinction between a platform and an architecture is important. A platform is a technology environment, such as a cloud analytics service. An architecture is the designed arrangement of technologies, data flows, controls and responsibilities that serves a business purpose.

2. Operational Databases and Analytical Systems

2.1 Relational databases

A relational database stores data in tables consisting of rows and columns. Relationships are represented through keys, and SQL is used for querying and manipulating data. Relational database management systems (RDBMSs) are especially suitable for structured, consistent transactions such as orders, payments and inventory updates.

Operational systems generally follow the principles of ACID transactions:

• Atomicity: a transaction succeeds completely or has no effect.
• Consistency: integrity rules remain satisfied.
• Isolation: concurrent transactions do not produce unacceptable interference.
• Durability: committed changes survive failure.

A banking transfer illustrates atomicity: deducting money from one account without crediting the other would be unacceptable. Relational systems support primary keys, foreign keys, constraints, indexes and transaction management. Their limitations may arise when data is semi-structured, horizontally scaled across many machines or queried through extremely large historical scans.

2.2 Online transaction processing and online analytical processing

Online transaction processing (OLTP) systems support frequent, short, concurrent updates. They are usually normalised to reduce duplication and protect update consistency. Online analytical processing (OLAP) systems support complex queries over historical data. They often use dimensional models containing fact tables and dimension tables.

A sales fact table might contain date, product, store, quantity and revenue keys. Dimension tables describe products, stores and dates. This arrangement improves analytical usability and can reduce the complexity of repeated business queries. A warehouse may also use columnar storage, which reads only the columns required by a query and often compresses repeated values effectively.

The same organisation may therefore require both an operational database and an analytical warehouse. Replicating every analytical query against the operational database can reduce application performance and create conflicting workloads.

3. Data Warehouses, Data Lakes and Lakehouses

3.1 Data warehouses

A data warehouse is a managed repository of integrated, historical and usually structured data designed for reporting and analysis. Data is commonly transformed before loading, a pattern known as extract, transform, load (ETL). Modern systems may also use extract, load, transform (ELT), in which raw or lightly processed data is loaded first and transformation occurs inside the analytical platform.

Strengths include reliable business definitions, strong SQL performance, access controls, auditing and consistent reporting. Weaknesses include the cost of modelling and transforming new sources, potential difficulty with unstructured data, and the risk that rigid designs delay experimentation.

3.2 Data lakes

A data lake stores data in its original or near-original form. It may contain structured tables, JSON documents, text, images, audio, application logs and sensor data. A lake commonly uses inexpensive and scalable object storage. Schema-on-read means that structure is interpreted when data is used rather than imposed fully when it is captured.

The lake approach supports data science and exploratory work, but flexibility creates risks. Without ownership, metadata, quality controls and retention rules, a lake can become a “data swamp”: data exists but cannot be trusted, located or understood. A mature lake requires cataloguing, lineage, classification, validation and lifecycle management.

3.3 Lakehouses

A lakehouse combines the low-cost, multi-format storage of a lake with warehouse-like management and performance. Important capabilities may include transactional updates on files, schema enforcement, versioning, governance, SQL access and support for both business intelligence and machine learning.

The term describes an architectural pattern rather than one universal product. Its success depends on open formats, reliable metadata, workload isolation, query optimisation and sound governance. A lakehouse does not automatically remove the need for data modelling or quality management.

4. Distributed Storage and Processing

A single server has limits in storage capacity, processing speed and fault tolerance. Distributed systems divide data and computation across multiple machines, or nodes. A cluster can increase capacity by adding nodes, a property called horizontal scaling.

4.1 Partitioning and replication

Partitioning divides a dataset into subsets. If records are partitioned by customer identifier, queries restricted to one customer may access only a subset. Poor partitioning can create data skew, in which one partition receives much more data or work than others.

Replication stores multiple copies of data. If a system maintains three replicas, the loss of one or even two nodes may not cause data loss, depending on the replication protocol and failure assumptions. Replication improves availability and read performance but increases storage consumption and coordination requirements.

A simplified storage estimate is:

The expression assumes that the replication factor applies uniformly and that compression is excluded. If logical data is 20 TB, the replication factor is 3 and overhead is estimated at 10% of replicated data, physical storage is approximately 20 × 3 × 1.10 = 66 TB. In practice, compression, erasure coding, temporary files and snapshots must also be considered.

4.2 Hadoop

Apache Hadoop historically provided four important ideas: distributed storage through the Hadoop Distributed File System (HDFS), cluster resource management through YARN, batch processing through MapReduce and a surrounding ecosystem of tools.

HDFS divides large files into blocks and distributes them across nodes, normally maintaining replicated copies. It is optimised for large sequential reads and writes rather than many small files or low-latency transactional updates. MapReduce expresses computation through a map stage, which processes input records, and a reduce stage, which aggregates or combines intermediate results. Between these stages, data is shuffled across the cluster. This model is robust for large batch workloads but can be inefficient for iterative algorithms because intermediate results may repeatedly be written to disk.

Hadoop remains historically important and some components remain in production, although cloud object storage and newer processing engines have changed many implementation choices.

4.3 Apache Spark

Apache Spark is a distributed processing engine designed for batch, interactive and streaming workloads. It provides APIs for SQL, DataFrames, machine learning and graph processing. Spark can cache intermediate data in memory when appropriate, reducing repeated disk access for iterative workloads.

Spark applications generally contain a driver and executors. The driver coordinates the application, constructs an execution plan and schedules tasks. Executors run tasks on worker nodes and may store cached data. Spark transformations are commonly lazy: the system builds a logical plan and executes it when an action requires results. This enables optimisation, but performance still depends on partitioning, joins, serialization, memory and data layout.

Spark is not itself a permanent database. It usually reads from and writes to storage systems such as object stores, HDFS, relational databases or lakehouse tables.

5. NoSQL Databases

NoSQL means “not only SQL” and refers to database families designed for flexible schemas, specialised access patterns or horizontal scale. They should not be regarded as universally superior to relational databases.

NoSQL systems often relax some relational assumptions to obtain availability, flexible modelling or scale. The choice should begin with workload requirements: read/write patterns, consistency needs, query types, latency, data relationships and failure tolerance.

A useful distinction is between strong consistency and eventual consistency. Strong consistency aims for reads to reflect the latest committed write according to the system’s guarantees. Eventual consistency permits temporary differences between replicas but assumes that, if updates stop, replicas eventually converge. Eventual consistency may be acceptable for social counters or product recommendations but inappropriate for certain financial balances.

The CAP theorem is commonly summarised as a distributed system facing a network partition cannot simultaneously guarantee both strong consistency and availability for all requests. CAP is not a menu in which a system permanently chooses only two properties; partition tolerance is generally necessary in a distributed network, and actual guarantees depend on the operation and implementation.

6. Batch, Micro-Batch and Streaming Architectures

6.1 Batch processing

Batch processing collects data and processes it periodically. Daily payroll, monthly financial consolidation and overnight demand forecasting are typical examples. Batch systems are often simpler, efficient for large scans and easier to reconcile. Their principal limitation is latency: decisions cannot use data that has not yet been processed.

6.2 Micro-batch processing

Micro-batch processing handles small groups of records at short intervals, such as every 30 seconds or five minutes. It offers much lower latency than daily batch while retaining some batch-like efficiency. It is useful when the business does not require record-by-record reaction.

6.3 Event streaming

In event streaming, records are processed continuously as events arrive. Producers publish events to topics, and consumers process them, often with message brokers such as Apache Kafka or cloud equivalents. A streaming architecture must address ordering, duplication, late events, retries, state management and failure recovery.

Important delivery concepts include:

• At-most-once: an event may be lost but is not normally processed twice.
• At-least-once: an event is not intentionally lost, but duplicates may occur.
• Exactly-once semantics: the system coordinates processing and output so that the intended effect occurs once within defined boundaries; this is a system-level guarantee, not a universal property of every external side effect.

Streaming systems often use event time, the time at which an event occurred, rather than processing time, the time at which the system received it. Windows group events for analysis. A tumbling window has fixed, non-overlapping intervals; a sliding window overlaps; a session window groups activity separated by periods of inactivity.

A useful architecture may combine streaming and batch. The streaming path gives rapid provisional results, while the batch path recomputes authoritative results and corrects late or invalid data. This pattern is sometimes called a lambda architecture, although modern lakehouse designs often seek a simpler unified processing approach.

7. Cloud Platforms and Architecture Choices

Cloud platforms provide elastic computing, managed storage, networking, identity services and specialised analytics products. The major service models are:

• Infrastructure as a service (IaaS): virtual machines, networks and storage are managed with substantial customer responsibility.
• Platform as a service (PaaS): the provider manages more of the runtime and infrastructure.
• Software as a service (SaaS): a complete application is delivered to users.

Cloud analytics commonly separates storage and compute. Data can remain in object storage while different clusters or serverless engines process it for different workloads. This improves flexibility but requires attention to data-transfer charges, access policies, workload isolation and performance variability.

Multi-cloud and hybrid-cloud strategies may improve resilience or satisfy regulatory requirements, but they increase integration, identity, skills and governance complexity. “Cloud” does not automatically mean secure, inexpensive or compliant. A sound decision evaluates total cost of ownership, including storage, compute, network transfer, support, licensing, migration, downtime and staff capability.

8. A Reference Architecture

Figure: A reference big-data architecture with governance applied across all layers.

The figure is a conceptual model, not a mandatory product stack. Data may move from sources through both batch and streaming ingestion. Storage may include an operational database, a warehouse and a lakehouse. Processing may produce curated tables, dashboards, alerts or machine-learning features. Governance must apply across the entire flow, not be added only at the end.

9. Applied Case: Retail Stock and Demand Intelligence

This is an illustrative case. A national retailer operates 400 stores and an online channel. Its transaction system records sales, but store managers often discover stock-outs only after customers complain. The organisation also has supplier files, promotion calendars, weather data and website events.

Problem

The existing relational system is reliable for sales transactions but is not designed for large historical queries, sensor-like website events or rapid cross-store forecasting. Reports are produced overnight, and data definitions differ between merchandising and finance.

Analysis

The retailer requires two distinct latency classes. Inventory transactions need strong consistency and immediate operational reliability. Demand monitoring should update every few minutes. Strategic forecasting can run hourly or daily. Data also varies from structured sales records to semi-structured web events and external files.

Approach

The retailer retains the relational OLTP database for orders and inventory updates. Change-data capture publishes selected changes to an event broker. Website events are streamed into cloud object storage and a lakehouse table. Supplier files are loaded in batch and validated. A warehouse provides governed finance and management reporting. Spark performs large-scale historical preparation and feature engineering, while a streaming processor calculates near-real-time stock-out indicators. A catalogue records owners, definitions, sensitivity classifications and lineage.

Access is role-based. Customer identifiers are tokenised for analytical use, and retention periods are defined for web events. Data-quality rules check duplicate orders, impossible quantities, missing store identifiers and late-arriving records. Forecast outputs are exposed through dashboards and replenishment applications rather than allowing managers to query raw data directly.

Outcome

Managers receive faster stock-out alerts, finance retains reconciled reporting, and analysts can experiment with historical and external data without burdening the transaction system. The architecture does not guarantee better decisions by itself: outcomes depend on forecast quality, supplier responsiveness, store processes and the willingness of users to act on trustworthy information.

Learning points

1. One system rarely serves every workload well.
2. Streaming and batch can coexist with different service objectives.
3. Governance is part of architecture, not administrative decoration.
4. The correct design follows business latency, consistency and regulatory requirements.

10. Governance, Security and Responsible Use

Big-data platforms increase the scale at which organisations can make decisions about people. Governance should address:

• Data ownership: named persons or teams are accountable for quality and appropriate use.
• Metadata and lineage: users can discover where data originated and how it was transformed.
• Quality: completeness, validity, uniqueness, timeliness and consistency are measured.
• Security: encryption, least-privilege access, network controls, key management and monitoring are implemented.
• Privacy: collection and processing have a legitimate purpose; sensitive data is minimised, protected and retained only as necessary.
• Regulatory compliance: requirements may include data-subject rights, breach reporting, sector rules, localisation and auditability.
• Responsible analytics: models are tested for bias, error, explainability and harmful consequences.

A lake containing personally identifiable information is not automatically acceptable merely because access is technically restricted. Purpose limitation, consent or another lawful basis, retention control and transparent communication remain important. Data provenance also matters for AI systems: undocumented training data can create reproducibility, copyright, privacy and bias problems.

11. Architecture Selection and Best Practices

A practical selection process asks:

1. What decisions or operational actions must the system support?
2. What latency is required: seconds, minutes, hours or days?
3. Is the workload transactional, analytical, exploratory or mixed?
4. What data formats, volumes, growth rates and quality problems exist?
5. What consistency, availability and recovery objectives apply?
6. Who owns the data and how will access be governed?
7. What are the full financial and skills costs?
8. How will success be measured?

Best practices include separating operational and analytical workloads, designing around access patterns, using open and documented formats where feasible, partitioning data carefully, monitoring pipelines, testing recovery, automating quality checks, documenting business definitions and controlling cloud costs. Small files can degrade distributed-storage performance; excessive partitioning can create metadata overhead; uncontrolled data duplication can increase both cost and privacy risk.

Architecture should also be evolutionary. Begin with a well-defined business use case and measurable service objectives. Avoid adopting Hadoop, Spark, a lakehouse or streaming merely because it is fashionable. A well-designed relational warehouse may be the best answer for a moderate, structured reporting problem.

Contemporary Developments

Current ecosystems increasingly combine cloud object storage, open table formats, serverless SQL, managed streaming, vector search and machine-learning services. Data contracts are gaining attention: producers document schemas, meanings, quality expectations and compatibility rules for consumers. Data mesh approaches emphasise domain ownership and data products, but they require strong standards and platform support; decentralisation without governance can multiply inconsistency.

Generative AI is increasing demand for governed document stores, retrieval pipelines, vector indexes and evaluation datasets. These technologies do not replace foundational architecture. Reliable identity, metadata, access control, quality and monitoring remain necessary whether the consumer is a dashboard, predictive model or language model.

Chapter Summary

Big-data architecture is an integrated arrangement of sources, ingestion, storage, processing, serving and governance. Relational databases are strong for structured transactions and integrity. Warehouses provide governed analytical performance, while lakes offer flexible, multi-format retention. Lakehouses seek to combine lake flexibility with warehouse management and reliability.

Distributed systems partition data and computation across nodes and use replication or other fault-tolerance methods. Hadoop established important distributed-storage and batch-processing patterns, while Spark provides a flexible engine for batch, interactive and streaming workloads. NoSQL databases support specialised access patterns and horizontal scale, but they require deliberate decisions about consistency, modelling and query design.

Batch processing is appropriate for periodic, high-throughput workloads; micro-batch reduces delay; streaming supports continuous decisions but introduces complexity around ordering, duplication, state and late events. Cloud platforms provide elasticity and managed services, but security, cost, skills and compliance remain organisational responsibilities. The strongest architecture is not the most complex one: it is the one that aligns technology with business latency, reliability, governance and value requirements.

Key Terms

Big-data ecosystem: Interconnected technologies, processes and responsibilities used to manage and analyse large or complex data.

OLTP: Online transaction processing for frequent, concurrent operational updates.

OLAP: Online analytical processing for complex queries over historical data.

ACID: Atomicity, consistency, isolation and durability properties of transactions.

Data warehouse: Governed, integrated repository designed primarily for analytical workloads.

Data lake: Scalable repository retaining data in diverse formats, often with schema-on-read.

Lakehouse: Architecture combining lake-style storage with warehouse-style management and analytics.

Partitioning: Dividing data or computation into subsets distributed across nodes.

Replication: Maintaining multiple copies of data for resilience or performance.

HDFS: Hadoop Distributed File System for distributed storage of large files.

MapReduce: Batch-processing model using map and reduce stages separated by a shuffle.

Apache Spark: Distributed engine supporting SQL, batch, machine learning and streaming.

NoSQL: Family of non-relational or not-only-SQL databases with specialised models and scale characteristics.

Event streaming: Continuous processing of events as they are produced.

Micro-batch: Processing small groups of records at short intervals.

Schema-on-write: Applying structure and validation before or during storage.

Schema-on-read: Interpreting structure when data is accessed.

Data lineage: Record of data origins, transformations and destinations.

Data governance: Policies, roles and controls that ensure data is trustworthy, secure and appropriately used.

Data mesh: Organisational and architectural approach emphasising domain-owned data products and federated governance.

Review Questions

5 Short Answer Questions

1. What is the primary difference between an OLTP system and an OLAP system?
2. Define schema-on-read and identify one environment in which it is useful.
3. What are partitioning and replication in a distributed system?
4. Distinguish batch processing from event streaming.
5. State two responsibilities of data governance.

5 Long Answer Questions

1. Compare databases, data warehouses, data lakes and lakehouses in terms of purpose, data structure, strengths and risks.
2. Explain how Hadoop and Spark approach distributed processing and identify circumstances in which each may be used.
3. Discuss the principal NoSQL models and explain why workload characteristics should guide database selection.
4. Explain the operational and governance challenges of a streaming architecture.
5. Evaluate the benefits and risks of separating storage and compute in a cloud analytics platform.

3 Analytical / Application Questions

1. A hospital needs highly reliable patient-record updates, daily management reports and near-real-time alerts for unusual device readings. Design a high-level architecture and justify the database, warehouse or lake, processing modes and governance controls you would select.
2. A retailer stores all data in a lake but analysts cannot determine which tables are accurate or current. Diagnose the causes of the data-swamp problem and propose a staged improvement plan.
3. An organisation is choosing between a relational warehouse and a lakehouse for structured finance data, raw customer documents and machine-learning experimentation. Develop a decision framework and explain whether a hybrid design may be preferable.

Multiple Choice Questions with Answers

Question 1

Which system is primarily designed for frequent, concurrent operational updates?

A. OLTP relational database

B. Data lake

C. Batch data warehouse

D. Graph visualisation tool

Correct Answer: A. OLTP relational database

Question 2

What does schema-on-read mean?

A. Data is deleted after it is queried

B. Structure is interpreted when data is accessed

C. Every record must satisfy a fixed schema before capture

D. Only relational data can be stored

Correct Answer: B. Structure is interpreted when data is accessed

Question 3

Which technology is principally a distributed processing engine rather than a permanent database?

A. Apache Spark

B. A relational database

C. A document database

D. A data warehouse

Correct Answer: A. Apache Spark

Question 4

A system processes events every five minutes in small groups. Which processing mode is this?

A. Pure OLTP

B. Micro-batch processing

C. Annual archival processing

D. Manual processing

Correct Answer: B. Micro-batch processing

Question 5

What is a major purpose of replication in distributed storage?

A. To remove all metadata

B. To maintain multiple copies for resilience or performance

C. To force every query to use one node

D. To convert unstructured data into SQL

Correct Answer: B. To maintain multiple copies for resilience or performance

Question 6

Which NoSQL model represents entities as nodes and their connections as relationships?

A. Key-value

B. Document

C. Graph

D. Wide-column only

Correct Answer: C. Graph

Question 7

A warehouse is preferred over a raw lake when the main requirement is which of the following?

A. Consistent, governed analytical reporting

B. Storing any file without metadata

C. Avoiding all data modelling

D. Processing only sensor events

Correct Answer: A. Consistent, governed analytical reporting

Question 8

Which statement about at-least-once event processing is correct?

A. Events may be duplicated and consumers should handle that possibility

B. Events are guaranteed to be lost

C. Every external side effect occurs exactly once automatically

D. Events cannot be retried

Correct Answer: A. Events may be duplicated and consumers should handle that possibility

Question 9

If 20 TB of logical data is stored with a replication factor of 3, before overhead the replicated storage is approximately:

A. 6.67 TB

B. 20 TB

C. 23 TB

D. 60 TB

Correct Answer: D. 60 TB

Question 10

Which practice most directly helps prevent a data lake from becoming a data swamp?

A. Removing all access controls

B. Adding cataloguing, ownership, lineage and quality rules

C. Storing only temporary files

D. Disabling retention policies

Correct Answer: B. Adding cataloguing, ownership, lineage and quality rules