Big Data, Cloud and Real-Time Analytics

Introduction

Data analytics has moved from analysing carefully structured, centrally stored datasets to processing continuously generated data at very large scale. Web applications, industrial sensors, mobile devices, financial transactions, connected vehicles and enterprise systems produce data whose volume, velocity, variety, veracity and value may exceed the capability of traditional database systems. This environment is commonly described as big data.

Big data is not defined only by size. A modest dataset can require big-data technologies if it arrives rapidly, has complex structure, must be processed with very low latency, or must be retained across geographically distributed systems. Modern analytics therefore combines distributed processing, scalable storage, cloud infrastructure and, increasingly, real-time stream processing.

The central architectural question is not “Which technology is most powerful?” It is: Which architecture satisfies analytical requirements at acceptable cost and risk? A batch system may be ideal for monthly regulatory reporting, whereas a streaming system may be necessary for fraud detection. A cloud data warehouse may suit governed business intelligence, while a data lake may be preferable for exploratory machine learning. Most contemporary organisations use a combination of these patterns.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the characteristics and architectural implications of big data.

2. Compare distributed batch, stream and hybrid processing models.

3. Explain scalable storage architectures, including data lakes, warehouses and lakehouses.

4. Evaluate cloud analytics services using performance, cost, resilience, security and governance criteria.

5. Calculate and interpret basic latency, throughput, storage and partitioning requirements.

6. Select an architecture for a given combination of volume, velocity, latency, cost and reliability requirements.

7. Critically assess operational, ethical, regulatory and environmental risks.

1. Big Data as an Architectural Problem

The traditional data-processing model assumes that data is collected, stored in a central database and periodically queried. This model remains effective for many workloads, but it becomes constrained when data generation and analytical demand increase faster than a single machine or tightly coupled system can scale.

The commonly used dimensions of big data are:

• Volume: the quantity of data stored or processed.
• Velocity: the rate at which data is generated, transmitted or processed.
• Variety: differences in formats, structures, semantics and sources.
• Veracity: uncertainty, noise, incompleteness and inconsistency.
• Value: the usefulness of data for decisions or operational outcomes.

These dimensions interact. A high-volume archive may tolerate hours of processing delay, while a low-volume safety signal may require a response within milliseconds. Consequently, architecture must be derived from workload requirements rather than from data size alone.

Important terminology

A workload is a defined pattern of data ingestion, storage, computation and access. Throughput is the amount of data or number of records processed per unit of time. Latency is the elapsed time between an event or request and the availability of a result. Scalability is the ability to handle increased workload. Elasticity is the ability to acquire and release resources as demand changes. Availability is the probability that a service is usable when required. Durability is the probability that stored data is not lost. These properties are related but not interchangeable: a system may be highly durable but temporarily unavailable.

2. Distributed Processing Principles

Distributed processing divides data and computation across multiple machines. A distributed system can scale beyond the physical limits of one server, but it introduces coordination, failure and data-placement problems.

2.1 Partitioning and parallelism

A dataset is commonly divided into partitions. Partitions may be created by hash, range, time interval, geographic region or another business key. Independent partitions can be processed in parallel.

If a workload contains \(N\) records and uses \(p\) parallel workers, an idealised processing time is:

\[

T_{ideal}=\frac{N}{p r}

\]

where \(r\) is the processing rate of one worker in records per second. This equation assumes balanced partitions, negligible communication, no failures and no shared bottleneck. Real processing time is greater because of data transfer, scheduling, serial work and skew.

A more realistic conceptual model is:

\[

T \approx T_{serial}+\frac{T_{parallel}}{p}+T_{communication}+T_{coordination}

\]

This is not an exact universal performance law; it is a design model showing why adding machines eventually produces diminishing returns. If one partition contains disproportionately more records than others, the slowest worker determines completion time. This is called data skew.

2.2 Data locality and network cost

Distributed frameworks attempt to execute computation close to the data, reducing network movement. Network transfer becomes especially expensive during a shuffle, in which records are redistributed between workers—for example, to group all transactions belonging to the same customer. Shuffles consume bandwidth, memory and disk and can dominate execution time.

Good design therefore uses suitable partition keys, filters data before aggregation, avoids unnecessary wide transformations and monitors skew. A partition key should distribute work evenly while supporting common access patterns. A key with only a few possible values can create hot partitions; a key with extremely high cardinality can create excessive management overhead.

2.3 Fault tolerance

Commodity distributed systems assume that component failure will occur. Fault tolerance may be provided through replication, checkpointing, recomputation, write-ahead logs or immutable input data. Replication improves availability and recovery but increases storage and network cost. Checkpoints reduce the amount of computation required after failure but introduce additional writes.

A useful distinction is between task-level recovery, where a failed computation is rerun, and state recovery, where a streaming application restores its accumulated state from a durable checkpoint. Recovery guarantees should be stated explicitly: at-most-once processing may lose records, at-least-once processing may produce duplicates, and exactly-once semantics generally require coordinated state, offsets and output transactions. “Exactly once” normally means exactly-once effects within a defined system boundary, not immunity from all downstream duplication.

3. Batch Analytics

Batch processing operates on a bounded dataset. The complete input is known, or treated as a finite snapshot, and the system computes a result over that input. Typical examples include payroll, monthly financial reporting, historical feature generation and annual risk analysis.

A batch job commonly follows this sequence:

1. Read files or database partitions.

2. Parse and validate records.

3. Filter, join or transform data.

4. Shuffle records when grouping or joining requires redistribution.

5. Aggregate or apply analytical functions.

6. Write results to a warehouse, report table or model repository.

Batch processing offers high throughput, reproducibility and efficient large-scale scans. It is often easier to test because the input snapshot can be retained and rerun. Its principal limitation is latency: a result may not be available until the next scheduled run.

Batch design considerations

Batch pipelines should be idempotent where possible. An idempotent job can be rerun without producing incorrect cumulative effects. For example, replacing a partition for a particular business date is safer than blindly appending the same records after a retry. Data quality checks should examine schema, completeness, valid ranges, uniqueness and referential relationships.

File size and format also affect performance. Columnar formats store values by column rather than by row and can reduce I/O when queries select only a subset of columns. Compression is often effective because adjacent values in a column tend to be similar. However, excessive numbers of small files create metadata and scheduling overhead, while very large files may reduce parallelism.

4. Stream and Real-Time Analytics

Stream processing handles an unbounded sequence of events. The system processes records as they arrive rather than waiting for a complete dataset. Use cases include payment fraud detection, equipment monitoring, personalised recommendations, cyber-security alerts and operational dashboards.

4.1 Event-time and processing-time

Processing time is when a system handles an event. Event time is when the event actually occurred. These can differ because of network delay, offline devices or retries. Analytics based only on processing time can assign events to the wrong time window.

A stream processor therefore uses watermarks, which are progress estimates indicating that events earlier than a certain event-time threshold are unlikely to arrive. Late events require a policy: discard them, update previous results, route them for correction or use a special reconciliation process. The appropriate policy depends on business risk and reporting requirements.

4.2 Windows and state

Because an unbounded stream cannot ordinarily be aggregated in its entirety, stream systems use windows. A tumbling window consists of adjacent, non-overlapping intervals. A sliding window has a fixed size and advances by a smaller step, so windows overlap. A session window groups events separated by no more than a specified inactivity gap.

For a count over a tumbling interval, the result for window \(w\) is:

\[

C_w=\sum_{i \in w} 1

\]

where each qualifying event contributes one and the window is defined by event-time boundaries. This simple expression depends on a correct event-time assignment and a stated late-event policy.

Stream applications frequently maintain state, such as a customer’s recent transaction total. State must be partitioned, checkpointed and recovered consistently. Unbounded state creates storage and performance risk; retention limits, time-to-live policies and approximate algorithms may be necessary.

4.3 Latency and throughput

A system’s end-to-end latency can be decomposed conceptually as:

\[

L_{end-to-end}=L_{ingest}+L_{queue}+L_{compute}+L_{output}

\]

where the terms represent ingestion, waiting, computation and result-publication delay. The equation is an accounting decomposition, not a claim that each component is constant. Queueing delay can increase sharply when arrival rate approaches service capacity.

Suppose events arrive at 2,000 records per second and a worker processes 500 records per second. At least four fully utilised workers are required for steady-state capacity, ignoring overhead:

\[

p \geq \left\lceil\frac{2000}{500}\right\rceil=4.

\]

In production, additional capacity is needed for skew, failures, checkpointing and traffic bursts. Designing exactly at the average rate risks growing queues and violating latency objectives.

5. Lambda, Kappa and Hybrid Architectures

A lambda architecture separates a batch layer, which computes comprehensive historical results, from a speed layer, which produces rapid approximate or incremental results. A serving layer combines the outputs. Its strength is the ability to recompute historical truth, but it duplicates logic and can produce inconsistencies between batch and speed implementations.

A kappa architecture treats the event log as the primary source and uses a stream-processing pipeline, replaying events when logic changes. It can reduce duplicated processing paths, but replaying a very large log may be expensive and stream logic must support historical correction.

A contemporary hybrid architecture often combines a durable event log, streaming transformations, periodic batch reconciliation and a governed analytical store. The choice should be based on recovery requirements, replay cost, acceptable temporary inconsistency and team capability—not on architectural fashion.

6. Scalable Storage Architectures

6.1 Data warehouses

A data warehouse is a governed analytical repository, usually optimised for structured data, SQL queries, historical analysis and controlled business definitions. Warehouses commonly use columnar storage, query optimisation, partition pruning and workload management. They are strong for dashboards, financial reporting and standardised metrics.

6.2 Data lakes

A data lake stores raw and processed data in economical object storage, often retaining structured, semi-structured and unstructured formats. It supports exploratory analytics and machine learning but requires strong cataloguing, metadata, access controls and lifecycle management. Without governance, a lake can become a “data swamp”: data exists but cannot be trusted or found.

6.3 Lakehouses

A lakehouse seeks to combine the low-cost, flexible storage of a lake with warehouse-like transactions, schema management, governance and performance. Table formats with transaction logs can support versioning, controlled updates and time travel over object storage. The architecture is attractive, but performance, interoperability and operational maturity vary by implementation.

6.4 Storage estimation

A preliminary storage estimate is:

\[

S_{raw}=R \times B \times D

\]

where \(R\) is records per second, \(B\) is average bytes per record and \(D\) is seconds of retention. If 10,000 records per second average 1,000 bytes for 30 days, then:

\[

S_{raw}=10{,}000 \times 1{,}000 \times (30\times86{,}400)

=25{,}920{,}000{,}000{,}000\text{ bytes},

\]

approximately 25.92 TB using decimal units. This excludes indexes, replicas, metadata, temporary files, compression effects and derived datasets. A capacity plan must include those factors explicitly rather than assuming that raw size is total cost.

7. Cloud Analytics Platforms

Cloud analytics platforms provide storage, compute, networking, managed databases, streaming services, security controls and monitoring through programmable interfaces. Their major advantage is not simply renting servers; it is the ability to provision resources rapidly and use managed services for complex operational functions.

Service models

• Infrastructure as a Service (IaaS): virtual machines, networks and storage are managed at a relatively low level.
• Platform as a Service (PaaS): the provider manages more of the runtime, scaling and operational platform.
• Software as a Service (SaaS): a complete application is consumed by users.
• Serverless analytics: users submit jobs or queries while the provider manages server allocation, often using consumption-based pricing.

Cloud architectures frequently separate storage and compute. This permits independent scaling: large historical data can remain in inexpensive object storage while compute clusters are started only when required. However, separation can increase network movement and query latency if data is repeatedly transferred.

Cloud selection criteria

A platform should be assessed using:

1. Functional fit: supported formats, SQL, streaming, machine learning and integration.

2. Performance: throughput, concurrency, predictable latency and scaling behaviour.

3. Cost: storage, compute, requests, data transfer, orchestration, support and idle resources.

4. Resilience: availability zones, backup, recovery objectives and regional continuity.

5. Security: identity, encryption, key management, isolation and auditability.

6. Governance: catalogues, lineage, retention, quality and policy enforcement.

7. Portability: open formats, export capability and dependence on proprietary services.

A useful financial distinction is between fixed costs, such as reserved capacity or platform administration, and variable costs, such as query execution, events, storage growth and egress. Total cost of ownership should also include engineering labour, migration, testing, observability and compliance.

8. Architecture Selection Framework

Architecture selection begins with a requirements profile:

• Expected average and peak ingestion rate.
• Data volume and retention period.
• Acceptable end-to-end latency.
• Query patterns and concurrency.
• Recovery time objective (RTO): maximum acceptable restoration time.
• Recovery point objective (RPO): maximum acceptable data loss measured in time.
• Consistency and correctness requirements.
• Security, privacy and jurisdictional constraints.
• Budget and staffing.

A practical decision sequence is:

1. Determine whether the workload is bounded, unbounded or both.

2. Classify outputs as operational, exploratory, regulatory or strategic.

3. Establish latency and freshness targets.

4. Identify replay, correction and audit requirements.

5. Estimate capacity at normal and peak load.

6. Compare managed and self-managed alternatives.

7. Design failure, backup and recovery behaviour.

8. Test with representative data and realistic concurrency.

For example, a daily sales report with a six-hour freshness target generally favours batch ingestion into a warehouse. A payment-authorisation decision requiring a response in less than one second favours a stream processor with low-latency state access. A manufacturing platform may require both: streaming alerts for immediate safety and batch analysis for maintenance planning.

9. Detailed Applied Case: Illustrative Retail Fraud Detection

Background

Consider a fictional national retailer processing card and online payments. It receives approximately 3,000 payment events per second during ordinary periods and up to 12,000 per second during promotional events. The business wants to flag suspicious transactions before authorisation is completed. It also needs daily investigations, model training and auditable decisions.

Problem

A nightly batch system identifies fraud too late. A direct query against the operational database increases transaction latency and risks disrupting sales. The organisation needs rapid scoring, historical analysis, resilience and controlled use of customer information.

Analysis

The workload has high velocity, burstiness and a sub-second decision target. It also has a bounded historical component for model training and investigation. Events may arrive late from mobile channels, and duplicate delivery is possible. Fraud rules require recent customer activity, device information and merchant-level patterns.

Approach

The proposed architecture uses:

1. An append-oriented event bus for payment and device events.

2. A stream processor that validates schemas, removes duplicates using event identifiers and maintains time-bounded customer and device state.

3. A low-latency feature store for current risk features.

4. A scoring service that applies rules and a machine-learning model.

5. An immutable raw event store for replay, investigation and audit.

6. A cloud warehouse or lakehouse for daily analysis and model development.

7. Batch reconciliation to correct late events and compare authorised outcomes with predictions.

8. Monitoring for throughput, lag, false-positive rate, model drift and feature freshness.

If the average event is 1,500 bytes and retention is 90 days at 3,000 events per second, raw storage is:

\[

3{,}000\times1{,}500\times(90\times86{,}400)

=34{,}992{,}000{,}000{,}000\text{ bytes},

\]

approximately 34.992 TB before replication and derived data. Capacity planning must use the 12,000-event peak for compute and queue design, not only the average rate.

Outcome

The streaming path supplies rapid provisional risk decisions, while the durable event store supports replay and audit. Batch reconciliation improves historical accuracy. The design accepts that a provisional decision may later be revised, but the business defines which decision points are irreversible and records model version, features, timestamp and reason codes.

Learning points

The case demonstrates that real-time analytics does not eliminate batch processing. It creates a layered architecture in which operational latency, historical completeness and governance are handled by specialised but coordinated components. It also shows why peak capacity, late data, duplicate events and auditability must be designed from the beginning.

10. Governance, Ethics and Security

Big-data systems can amplify errors at scale. Poor-quality source data may produce systematically incorrect decisions. Predictive models can reproduce historical discrimination, especially when sensitive characteristics are directly or indirectly encoded. Organisations should document data provenance, lawful purpose, retention, access rights and model limitations.

Important controls include data minimisation, purpose limitation, encryption in transit and at rest, strong identity management, least privilege, separation of duties, masking or tokenisation of sensitive fields, and immutable audit logs. Privacy regulations may impose requirements concerning consent, lawful processing, data subject rights, cross-border transfer and automated decision-making. Exact obligations depend on jurisdiction and sector; technical teams must work with legal and compliance specialists.

Streaming systems create additional governance issues. Events may be copied into logs, caches, feature stores and analytical repositories, making deletion and retention enforcement difficult. Data contracts should specify ownership, schema, quality expectations, compatibility rules and retention. Access should be granted to required fields rather than entire datasets.

11. Challenges and Limitations

Distributed and cloud analytics offer scale but introduce significant risks:

• Operational complexity: many services, interfaces and failure modes must be managed.
• Cost unpredictability: unbounded queries, data egress and idle resources can cause unexpected bills.
• Vendor lock-in: proprietary formats and APIs may make migration difficult.
• Data quality: faster processing does not make inaccurate data reliable.
• Latency trade-offs: stronger consistency, encryption, checkpointing and cross-region replication may add delay.
• Skew and hotspots: uneven keys can overload individual workers.
• Observability gaps: aggregate success metrics can conceal late data or silent quality degradation.
• Environmental impact: energy and material use should be considered when selecting retention, replication and compute strategies.

The appropriate response is not to eliminate all complexity but to make assumptions explicit, automate controls and test realistic failure conditions.

12. Contemporary Developments and Best Practices

Current practice increasingly emphasises open table formats, data products, real-time machine-learning features, vector and multimodal data, serverless execution, confidential computing and policy-driven governance. Generative AI can assist with query development, anomaly investigation and documentation, but generated code and explanations require testing, access control and human review. AI does not remove the need for reliable data pipelines.

Best practices include:

• Define service-level objectives for freshness, latency, availability and correctness.
• Separate raw, validated and curated data layers.
• Use schema evolution rules and data contracts.
• Design for replay, idempotency and duplicate handling.
• Partition according to workload while monitoring skew.
• Prefer incremental processing where full rescans are unnecessary.
• Apply lifecycle policies, compression and deletion controls.
• Automate infrastructure, testing, lineage and quality monitoring.
• Load-test peak conditions and deliberately test failures.
• Record model versions, feature definitions and decision explanations.
• Review total cost and environmental impact regularly.

Chapter Summary

Big-data analytics is an architectural discipline concerned with distributed computation, scalable storage, cloud economics and timely decision-making. Batch processing provides efficient, reproducible analysis of bounded datasets, while stream processing supports continuous data and low-latency responses. Event time, windows, watermarks, state and late-data policies are essential to correct streaming results.

Distributed systems achieve scale through partitioning and parallelism, but communication, skew, coordination and failure reduce ideal performance. Data warehouses provide governed analytical access; data lakes offer flexible, economical storage; and lakehouses seek to combine these capabilities. Cloud platforms add elasticity and managed services, but they require careful control of cost, security, portability and resilience.

Architecture selection should begin with volume, velocity, latency, peak load, retention, consistency, RTO, RPO, governance and budget. Hybrid designs are often appropriate because real-time operational decisions and historical analytical truth have different requirements. Ethical, privacy and regulatory controls must be embedded throughout the data lifecycle.

Key Terms

Big data: Data whose scale, speed, diversity or complexity requires specialised analytical architectures.

Batch processing: Processing a bounded dataset as a scheduled or finite computation.

Stream processing: Continuous processing of an unbounded sequence of events.

Throughput: Amount of data or number of records processed per unit of time.

Latency: Time between an event or request and the availability of its result.

Partitioning: Dividing data into segments for storage, management or parallel processing.

Data skew: Uneven distribution of records or work across partitions.

Shuffle: Redistribution of records between distributed workers.

Event time: Time at which an event actually occurred.

Processing time: Time at which a system processes an event.

Watermark: Progress estimate used to manage likely late events.

Window: A defined subset of a stream used for computation.

Data warehouse: Governed analytical repository optimised for structured queries.

Data lake: Flexible storage repository for raw and processed data in multiple formats.

Lakehouse: Architecture combining lake storage with warehouse-like management and performance features.

Elasticity: Ability to acquire and release resources in response to demand.

RTO: Maximum acceptable time to restore a service after disruption.

RPO: Maximum acceptable period of data loss after disruption.

Idempotency: Property whereby repeating an operation produces the same intended result.

Data contract: Agreement specifying data schema, quality, ownership and compatibility expectations.

Review Questions

Short Answer Questions

1. Distinguish throughput from latency.

2. Why can data skew reduce distributed-processing performance?

3. What is the difference between event time and processing time?

4. State one advantage and one limitation of a data lake.

5. Define RTO and RPO.

Long Answer Questions

1. Compare batch and stream processing with respect to input, latency, correctness, testing and typical applications.

2. Explain how partitioning, shuffling and data locality affect distributed computation.

3. Compare data warehouses, data lakes and lakehouses.

4. Discuss the principal criteria for selecting a cloud analytics platform.

5. Explain why a hybrid architecture may be preferable to a purely batch or purely streaming architecture.

Analytical / Application Questions

1. A system receives 8,000 events per second, and one worker processes 400 events per second. Calculate the minimum number of workers required for steady-state processing, ignoring overhead, and explain why production capacity should be higher.

2. Design an architecture for equipment-safety monitoring requiring alerts within two seconds and monthly maintenance analysis. Justify storage, processing and governance choices.

3. A company stores 5,000 records per second, each averaging 800 bytes, for 60 days. Estimate raw decimal storage and identify at least four additional capacity factors that must be considered.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly distinguishes a streaming workload from a batch workload?

A. The data is always unstructured

B. The input is treated as an ongoing, unbounded sequence

C. The workload never requires storage

D. The workload cannot use SQL

Correct Answer: B. The input is treated as an ongoing, unbounded sequence

Question 2

What is a shuffle in distributed processing?

A. Compressing every output file

B. Deleting duplicate records

C. Redistributing records among workers

D. Encrypting data at rest

Correct Answer: C. Redistributing records among workers

Question 3

A worker processes 500 records per second and events arrive at 2,000 records per second. Ignoring overhead, what is the minimum number of workers for steady-state capacity?

A. 2

B. 3

C. 4

D. 5

Correct Answer: C. 4

Question 4

Which mechanism helps a stream processor reason about late event-time data?

A. Watermark

B. Primary key

C. Compiler

D. Snapshot isolation

Correct Answer: A. Watermark

Question 5

Which architecture is primarily optimised for governed structured analytical queries?

A. Data warehouse

B. Event bus

C. Operational cache

D. Raw device registry

Correct Answer: A. Data warehouse

Question 6

What is the principal purpose of partitioning data in a distributed system?

A. To prevent all parallel execution

B. To divide work and data for scalable processing

C. To remove the need for metadata

D. To guarantee zero latency

Correct Answer: B. To divide work and data for scalable processing

Question 7

Which statement about RPO is correct?

A. It specifies maximum acceptable service restoration time

B. It specifies maximum acceptable data loss measured in time

C. It specifies query throughput

D. It specifies encryption strength

Correct Answer: B. It specifies maximum acceptable data loss measured in time

Question 8

Why is idempotency valuable in data pipelines?

A. It makes repeated execution produce the same intended result

B. It guarantees that no machine will fail

C. It eliminates the need for storage

D. It prevents all late events

Correct Answer: A. It makes repeated execution produce the same intended result

Question 9

Which cloud property allows resources to be acquired and released as demand changes?

A. Durability

B. Elasticity

C. Veracity

D. Serialisability

Correct Answer: B. Elasticity

Question 10

A lakehouse generally seeks to combine which capabilities?

A. Only transactional processing and no analytics

B. Low-cost flexible lake storage with warehouse-like management

C. Streaming without persistence

D. Manual reporting without governance

Correct Answer: B. Low-cost flexible lake storage with warehouse-like management