4. Big-Data Storage, Processing and Platform Technologies

Introduction and Context

Big-data analytics depends not only on algorithms but also on the platforms that store, move, process, and govern data. A sophisticated predictive model cannot compensate for an unsuitable storage engine, excessive data movement, weak fault tolerance, or an architecture that cannot meet latency and cost requirements. Storage and processing choices therefore constitute architectural decisions rather than merely implementation details.

The central design problem is that big-data workloads are heterogeneous. A transactional application requires consistency and rapid point updates; a data lake must retain diverse files at large scale; a recommendation engine may require millisecond graph or key-value lookups; and a regulatory reporting system may demand reproducible batch computation. No single technology optimally satisfies all of these requirements. Contemporary architectures consequently combine relational databases, NoSQL systems, object stores, analytical warehouses, stream processors, containers, orchestration platforms, and serverless services.

This chapter examines these technologies comparatively. It treats platform selection as a multi-dimensional decision involving data model, workload pattern, latency, consistency, elasticity, resilience, security, governance, operational capability, and total cost of ownership.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Compare relational, NoSQL, graph, columnar, and object storage models.

2. Distinguish batch, stream, micro-batch, and interactive processing.

3. Explain the roles of distributed processing engines and cloud-native services.

4. Evaluate containers, orchestration, and serverless computing for scalable workloads.

5. Apply a structured framework to platform selection.

6. Identify architectural, operational, ethical, and governance risks.

7. Design and critically assess a multi-platform big-data architecture.

From Data Repositories to Data Platforms

A storage system answers where and how data is persisted; a processing engine answers how computation is executed; a platform integrates storage, compute, networking, security, monitoring, and governance. The distinction matters because many modern services combine these functions. A cloud data warehouse, for example, may provide columnar storage, distributed query execution, workload management, and access control as one managed service.

A useful architectural distinction is between systems of record, systems of insight, and systems of engagement. A system of record preserves authoritative operational data. A system of insight transforms historical and real-time data for analysis. A system of engagement delivers results to users or automated decisions. Data is often copied or streamed among these systems, creating the need for lineage, schema management, quality controls, and explicit ownership.

The data lifecycle normally includes ingestion, validation, persistence, transformation, serving, archival, and deletion. Each stage creates different requirements. Raw data benefits from inexpensive scalable storage, whereas serving applications need indexed structures and predictable latency. Treating the entire lifecycle as one undifferentiated database frequently produces unnecessary cost and poor performance.

Storage Models and Their Design Assumptions

Relational storage

Relational databases represent data in tables containing rows and attributes, with relationships expressed through keys. SQL provides declarative querying, joins, constraints, transactions, and mature optimization. The relational model is particularly effective where data is structured, relationships are stable, and correctness of multi-record transactions is essential.

The traditional ACID properties are atomicity, consistency, isolation, and durability. Horizontal scaling can be achieved through partitioning, replication, distributed SQL, or sharding, but cross-partition transactions and joins may become expensive. Relational systems remain important in big-data architectures because large-scale analytics does not eliminate the need for reliable operational transactions.

NoSQL storage

NoSQL is an umbrella term for non-relational systems designed to relax some relational assumptions in favor of flexible schemas, horizontal scalability, or specialized access patterns. Major forms include key-value, document, wide-column, and time-series databases.

Key-value stores are efficient for direct lookup by a known key, such as sessions or cached recommendations. Document databases store semi-structured objects, often represented as JSON, and suit evolving application records. Wide-column systems organize data by row keys and column families and are useful for high-volume distributed writes. Time-series systems optimize timestamped measurements and temporal queries.

NoSQL design generally begins with query patterns rather than normalization. Data may be denormalized and duplicated to avoid distributed joins. This improves read performance but increases update complexity and the risk of inconsistent copies. Many systems offer tunable or eventual consistency, while others provide stronger guarantees for selected operations. The CAP perspective highlights a trade-off in the presence of a network partition between broad availability and strong consistency; it should not be interpreted as a simple choice made independently of workload and implementation.

Graph storage

Graph databases represent entities as nodes and relationships as edges, often with properties attached to both. They are appropriate when relationship traversal is central to the workload: fraud rings, knowledge graphs, supply-chain dependencies, identity resolution, and recommendation networks are common examples. Graph query languages support pattern matching and path-oriented operations.

Graph systems are not automatically superior for all connected data. Massive aggregations over simple event records may be more efficient in columnar analytical systems. Graph technology is selected when the topology and traversal logic carry analytical meaning and would otherwise require repeated, expensive joins.

Columnar storage and analytical warehouses

Columnar systems store values from the same attribute together rather than storing complete rows contiguously. Analytical queries commonly scan a small number of columns over many rows, so columnar storage reduces I/O. Compression is also effective because adjacent values often have similar types and distributions. Vectorized execution, predicate pushdown, partition pruning, and parallel query processing further improve performance.

Columnar storage is optimized for read-heavy analytical workloads, not necessarily for frequent single-row updates. Cloud warehouses often separate compute from storage, allowing independent scaling and elastic query clusters. Nevertheless, poorly designed queries, uncontrolled concurrency, data duplication, and inefficient partitioning can produce high cost.

Object storage and data lakes

Object storage organizes data as objects within buckets or containers, identified by keys and accompanied by metadata. It provides high durability, broad scalability, and low-cost persistence for files such as Parquet, ORC, JSON, images, and machine-learning artifacts. It is a foundational layer for data lakes.

Object storage is not equivalent to a database. It usually lacks transactional row updates, low-latency indexing, and arbitrary record-level queries. Performance depends on file format, object size, partitioning, metadata management, and the processing engine. Columnar open formats such as Parquet make object stores suitable for analytical processing, while table formats add transaction-like semantics, schema evolution, versioning, and time travel. A data lakehouse combines inexpensive open storage with warehouse-like management and query behavior, although its guarantees depend on implementation.

Processing Paradigms

Batch processing

Batch processing operates on bounded datasets. It is suitable for historical aggregation, feature generation, periodic billing, regulatory reports, and large-scale transformation. Batch jobs maximize throughput and can tolerate latency ranging from minutes to hours. Distributed engines divide data into partitions and execute tasks in parallel, recovering failed tasks through recomputation or intermediate data management.

MapReduce established a durable conceptual pattern: map records into intermediate key-value pairs and reduce values associated with each key. Modern engines often retain this partitioned, parallel model while improving performance through directed acyclic graphs, in-memory execution, query optimization, and vectorization. Spark, for example, supports batch, SQL, machine learning, and streaming workloads through a common ecosystem; other engines specialize in SQL, massively parallel data warehousing, or large-scale transformation.

Stream processing

Stream processing analyzes unbounded sequences of events as they arrive. Applications include fraud detection, equipment monitoring, personalization, logistics, and operational alerting. Important concepts include event time, processing time, windows, watermarks, partitioning, state, replay, and delivery semantics.

A tumbling window is non-overlapping; a sliding window advances at intervals and overlaps; a session window groups activity separated by periods of inactivity. Event-time processing is valuable when events arrive late or out of order. Watermarks express a system’s progress expectation and help balance completeness against latency. Stateful operators require durable state and recovery mechanisms.

At-most-once processing may lose records but avoids duplicates; at-least-once processing may repeat records and therefore requires idempotent consumers or deduplication; exactly-once claims are meaningful only within defined system boundaries and usually require coordinated state, checkpoints, and transactional sinks. Streaming systems should not be selected solely because data is described as “real time”; the business value of lower latency must justify its additional operational complexity.

Micro-batch and unified processing

Micro-batch systems process small groups of events at short intervals, offering a compromise between pure streaming and batch. Unified engines permit the same transformations to run over historical files and live streams, reducing code duplication. However, semantic differences remain: late data, changing schemas, replay, and state retention require explicit design even when APIs look similar.

Cloud-Native Platform Technologies

Cloud-native architectures use elastic infrastructure, managed services, automation, and failure-aware design. Managed object storage, cloud warehouses, message brokers, stream services, distributed databases, and machine-learning platforms reduce undifferentiated operational work. They do not remove responsibility: organizations still manage identity, data classification, configuration, cost, quality, and regulatory compliance.

Decoupling storage from compute allows several processing engines to operate on the same governed data. Elastic compute can be provisioned for a workload and released afterward. Nevertheless, data transfer, repeated scans, service-specific formats, and network egress can undermine the expected benefits. Portability also varies: open file formats and standard SQL may improve portability, while proprietary APIs and tightly integrated services may increase switching costs.

Containers

A container packages an application and its dependencies while sharing the host operating system kernel. Containers improve reproducibility, deployment speed, isolation, and environment consistency. They are useful for custom stream processors, data-quality services, feature pipelines, and APIs that cannot be expressed through a managed service.

Containers do not provide complete virtual-machine isolation, and containerizing a poorly designed application does not make it scalable. Images require vulnerability scanning, minimal base layers, signed provenance, secret separation, and controlled registries.

Orchestration

Orchestration platforms schedule containers, manage service discovery, support rolling deployment, restart failed workloads, and scale replicas. Kubernetes is a prominent example. In data platforms, orchestration may coordinate ingestion, transformation, model training, and serving, while workflow systems express dependencies and retries.

Orchestration introduces a control-plane learning curve. Teams must understand resource requests and limits, persistent volumes, network policies, observability, upgrades, and failure domains. It is justified when workload diversity, portability, or operational control outweighs the cost of managing the platform.

Serverless computing

Serverless services abstract server provisioning and commonly charge according to invocations, execution time, or resource consumption. They are effective for event-triggered transformations, lightweight APIs, scheduled quality checks, and bursty workloads. Benefits include rapid deployment and automatic scaling.

Limitations include startup latency, execution duration limits, concurrency controls, opaque infrastructure behavior, vendor dependence, and difficulty with large stateful jobs. Serverless is therefore a workload characteristic, not a universal architectural destination. Long-running distributed transformations usually require a managed cluster, warehouse, or specialized processing service.

A Framework for Platform Selection

Platform selection should begin with requirements rather than product popularity. A practical decision sequence is:

1. Characterize data: volume, velocity, variety, structure, retention, sensitivity, and relationship density.

2. Characterize access: point lookup, transactional update, graph traversal, full scan, aggregation, search, or event subscription.

3. Specify service levels: latency, throughput, freshness, availability, recovery point objective, and recovery time objective.

4. Choose consistency and processing semantics: ACID, eventual consistency, ordering, replay, idempotence, and delivery guarantees.

5. Evaluate scale and failure behavior: partitioning, replication, elasticity, hotspots, backpressure, and disaster recovery.

6. Assess organizational constraints: skills, support model, regulatory location, integration, portability, and procurement.

7. Estimate total cost: storage, compute, network, licenses, operations, observability, migration, and idle capacity.

8. Test empirically: use representative data, realistic concurrency, failure injection, and production-like queries.

A weighted decision matrix can formalize the comparison, but numerical scores should not conceal critical constraints. A platform that fails a mandatory security or latency requirement cannot be rescued by a high score for convenience. Architecture review should also consider the number of technologies the organization can operate competently; polyglot persistence is useful, but uncontrolled technology proliferation increases failure and governance risk.

Applied Case: A National Retail Fraud-Analytics Platform

Background

The following is an illustrative case. A national retailer operates thousands of stores and an online channel. It records purchases, payment attempts, device signals, account activity, and product relationships. Fraud analysts need historical investigations, while the checkout service requires a decision within a few hundred milliseconds.

Situation and problem

The existing system stores all records in a relational database. It provides reliable transactions but suffers from analytical contention, slow relationship queries, and limited ability to process event streams. A proposal recommends replacing it with a single NoSQL database. The architecture group rejects this binary choice and analyzes workload separation.

Analysis

Operational orders and payments require transactional integrity and remain in a relational system. Events are published to a durable message broker and retained in object storage using partitioned columnar files. A stream processor computes velocity features, such as repeated payment attempts or unusual device-location combinations, and writes current features to a low-latency key-value store. A graph store maintains selected account, device, and merchant relationships for investigation and high-value pattern analysis. Historical aggregates are queried in a columnar warehouse.

The design separates authoritative transactions from analytical copies. It also introduces risks: duplicated data, delayed propagation, false positives, privacy concerns, and operational complexity. Data contracts define event schemas, and identifiers are tokenized where possible. Feature calculations are designed to be idempotent so replay does not inflate counts.

Approach and solution

Containers package custom feature services and are orchestrated for controlled deployment. Managed stream and warehouse services handle elastic demand. Serverless functions perform lightweight enrichment and notification, but not the core stateful feature computation. Monitoring measures event lag, decision latency, duplicate rates, feature freshness, cost, and model drift.

Outcome

The resulting platform supports fast decisions without subjecting the transactional database to analytical scans. Investigators gain historical and relationship-oriented views, while object storage provides a replayable source for new features. The organization accepts that no single database is optimal and governs the data flows as carefully as the individual technologies.

Learning points

The case demonstrates workload-oriented architecture, the value of event replay, the necessity of idempotence, and the importance of measuring business service levels rather than merely database benchmark scores. It also shows that polyglot architecture should be justified by distinct access patterns, not by technological fashion.

Risks, Governance, and Professional Responsibilities

Distributed platforms create failure modes that are easy to overlook. Partial failure, network partitions, clock differences, duplicate events, schema drift, skewed partitions, hot keys, backpressure, and unbounded state can degrade correctness or availability. Data lakes can become “data swamps” when ownership, cataloguing, quality, and retention are absent. Over-partitioned files create metadata overhead, whereas oversized files can reduce parallelism.

Security must be designed across storage and processing layers. Controls include least-privilege identity, encryption in transit and at rest, network segmentation, key management, secrets protection, immutable audit logs, vulnerability management, and tested recovery. Sensitive attributes require classification, minimization, masking, retention limits, and controlled secondary use. Regulatory duties may include lawful processing, access and deletion rights, residency restrictions, and demonstrable accountability, depending on jurisdiction.

Analytical platforms also influence individuals. Fraud systems can produce discriminatory error rates; location and behavioral data can enable intrusive surveillance; and automated decisions may be difficult to explain. Governance should therefore address purpose limitation, human review, model and feature lineage, bias evaluation, appeal processes, and separation of duties. Technical convenience is not a sufficient justification for retaining every event indefinitely.

Current Developments

Several developments are reshaping platform architecture. Lakehouse table formats seek to provide transactional updates and schema governance over object storage. Open table and file formats improve interoperability, although engines may implement features differently. Data mesh approaches emphasize domain ownership and data products, but they require strong federated governance and platform support. Data fabric architectures emphasize metadata-driven integration and policy enforcement across distributed sources.

Streaming SQL, change-data capture, vectorized execution, and increasingly autonomous data operations are reducing the distance between operational events and analytical decisions. AI assistants can help generate queries, documentation, and pipeline code, but generated artifacts require testing, access review, and provenance controls. Hardware acceleration and specialized processing may improve performance for particular workloads, yet benchmarking remains essential because gains depend on data shape and query behavior.

Best-Practice Principles

A durable big-data platform generally follows these principles:

• Start with workload and service-level requirements, not a preferred product.
• Separate systems of record from analytical and serving workloads when contention is likely.
• Prefer open, well-documented formats where portability and long-term access matter.
• Design partitioning, keys, schemas, and indexes from actual query patterns.
• Make pipelines replayable, observable, idempotent, and explicitly governed.
• Treat latency, correctness, availability, security, and cost as jointly optimized objectives.
• Use managed services when they reduce undifferentiated work, but evaluate lock-in and failure boundaries.
• Test with realistic data, concurrency, skew, late events, and injected failures.
• Limit technology diversity to capabilities the organization can secure and operate.
• Document ownership, lineage, retention, recovery, and decommissioning decisions.

Chapter Summary

Big-data architecture is an exercise in matching storage and processing semantics to workload requirements. Relational systems provide strong transactional guarantees; NoSQL systems support flexible, distributed access patterns; graph stores optimize relationship traversal; columnar systems accelerate analytical scans; and object storage offers scalable, economical persistence for diverse files. Batch processing emphasizes throughput, while stream processing addresses unbounded events, state, time, and latency.

Cloud-native services, containers, orchestration, and serverless computing provide different levels of abstraction and operational control. None is universally superior. Platform selection should evaluate data characteristics, access patterns, service levels, consistency, resilience, security, governance, skills, portability, and total cost. Effective architectures often combine several technologies, provided that data contracts, lineage, observability, replay, idempotence, and accountability are deliberately designed.

Key Terms

Relational database: A table-based system using keys, constraints, and declarative queries.

ACID: Atomicity, consistency, isolation, and durability properties of reliable transactions.

NoSQL: A family of non-relational systems optimized for flexible schemas or distributed access patterns.

Key-value store: A database that retrieves values through unique keys.

Document database: A system that stores semi-structured records as documents.

Wide-column store: A distributed database organized around row keys and column families.

Graph database: A system representing entities and relationships as nodes and edges.

Columnar storage: Storage that groups values by column to improve analytical scans and compression.

Object storage: Scalable file-like storage based on objects, keys, and metadata.

Data lake: A repository retaining diverse data, commonly on object storage, for later processing.

Lakehouse: An architecture combining open lake storage with warehouse-style management and transactions.

Batch processing: Computation over a bounded collection of data.

Stream processing: Computation over continuously arriving, unbounded events.

Event time: The time at which an event actually occurred.

Watermark: A progress indicator used to manage late events in stream processing.

Idempotence: The property that repeating an operation produces the same intended result.

Container: A packaged application and its dependencies sharing an operating-system kernel.

Orchestration: Automated scheduling, scaling, networking, and lifecycle management of workloads.

Serverless computing: Execution in which infrastructure provisioning is largely abstracted from the user.

Polyglot persistence: Use of multiple storage technologies selected for different workload requirements.

Review Questions

Short Answer Questions

1. Why does a relational database remain valuable in a big-data architecture?

2. What access pattern is particularly suitable for a graph database?

3. Explain the difference between event time and processing time.

4. What problem does columnar storage address in analytical workloads?

5. Define idempotence in the context of data pipelines.

Long Answer Questions

1. Compare relational, NoSQL, graph, columnar, and object storage with respect to data model, scalability, consistency, and workload suitability.

2. Explain batch, micro-batch, and stream processing, including windows, watermarks, state, and delivery semantics.

3. Discuss the benefits and risks of separating storage from compute in cloud-native platforms.

4. Evaluate containers, orchestration, and serverless computing as alternatives for deploying big-data workloads.

5. Develop a platform-selection framework that includes technical, organizational, governance, and economic criteria.

Analytical / Application Questions

1. Design a storage and processing architecture for a smart-factory system that must retain sensor history, detect failures within seconds, and support annual engineering analysis. Justify each platform choice.

2. An organization proposes migrating every workload to a single document database. Critically assess the proposal using workload patterns, consistency, analytics, governance, and total cost.

3. A stream-processing system produces duplicate fraud alerts after recovery. Analyze the likely causes and propose changes involving delivery semantics, state, idempotence, and sink design.

22. Multiple Choice Questions with Answers

Question 1

A retailer needs millisecond retrieval of a customer’s current session by a known identifier. Which storage model is most directly suited to this access pattern?

A. Key-value store

B. Graph database

C. Columnar warehouse

D. Object storage

Correct Answer: A. Key-value store

Question 2

Why does columnar storage commonly improve analytical query performance?

A. It guarantees multi-record ACID transactions

B. It reads and compresses only the required attributes efficiently

C. It eliminates the need for partitioning

D. It stores every relationship as an edge

Correct Answer: B. It reads and compresses only the required attributes efficiently

Question 3

A stream-processing application must handle events that arrive late and out of order. Which combination is most relevant?

A. Foreign keys and normalization

B. Object keys and file compaction

C. Event time and watermarks

D. Container images and rolling updates

Correct Answer: C. Event time and watermarks

Question 4

Which architecture best separates transactional correctness from large-scale historical analysis?

A. Running analytical scans directly on the payment database

B. Using only a graph database for every workload

C. Publishing events to analytical storage while retaining the relational system of record

D. Replacing all persistent storage with serverless functions

Correct Answer: C. Publishing events to analytical storage while retaining the relational system of record

Question 5

What is the principal purpose of idempotence in a replayable data pipeline?

A. To ensure repeated processing does not create unintended duplicate effects

B. To increase the number of distinct schemas automatically

C. To prevent all network partitions

D. To guarantee that every query is interactive

Correct Answer: A. To ensure repeated processing does not create unintended duplicate effects

Question 6

Which workload is generally the strongest candidate for serverless execution?

A. A continuously running, stateful distributed transformation with large retained state

B. A bursty event-triggered enrichment function with short execution time

C. A high-volume relational transaction requiring coordinated multi-row updates

D. A graph traversal requiring a permanently resident graph index

Correct Answer: B. A bursty event-triggered enrichment function with short execution time

Question 7

What is a major architectural risk of denormalizing data in a NoSQL system?

A. Analytical scans become impossible in every case

B. Duplicated values may become inconsistent during updates

C. The system cannot scale horizontally

D. All transactions automatically become serializable

Correct Answer: B. Duplicated values may become inconsistent during updates

Question 8

Which factor should be treated as a mandatory constraint rather than merely a weighted preference in platform selection?

A. A team’s familiarity with a fashionable tool

B. A product’s attractive user interface

C. Failure to meet a required regulatory or latency condition

D. The number of optional dashboard templates

Correct Answer: C. Failure to meet a required regulatory or latency condition

Question 9

What does orchestration primarily provide for containerized workloads?

A. Automatic replacement of all application logic

B. Scheduling, scaling, service discovery, and lifecycle management

C. Guaranteed exactly-once semantics for every data sink

D. Permanent elimination of vendor dependence

Correct Answer: B. Scheduling, scaling, service discovery, and lifecycle management

Question 10

In the illustrative fraud platform, why is a message stream and object storage used alongside the relational system?

A. To remove the need for transaction processing

B. To provide replayable analytical data and decouple downstream processing

C. To ensure that no data is ever duplicated

D. To make graph traversal unnecessary

Correct Answer: B. To provide replayable analytical data and decouple downstream processing