Big Data and Real-Time Data Engineering

Introduction and Context

Modern information systems generate data continuously from transactions, mobile devices, sensors, applications, industrial equipment, and external services. The scale, speed, and diversity of these data exceed the assumptions of traditional database systems. Big data engineering addresses the design of platforms and processes that store, transform, govern, and analyse large datasets, while real-time data engineering addresses systems in which data must be processed shortly after it is produced.

The distinction between batch and streaming is useful but not absolute. Batch processing operates on a bounded collection of data, usually according to a schedule. Stream processing operates on an unbounded sequence of events. Contemporary architectures often combine both: historical data is processed in batch, while new events are processed incrementally. The objective is not merely low latency. A professionally designed system must also provide correctness, durability, scalability, observability, security, and an explicit response to late, duplicated, or missing data.

This chapter develops the principles behind distributed storage and computation, event sourcing, change data capture (CDC), stream-processing semantics, data contracts, and high-volume architecture. It also examines trade-offs that arise when organisations seek both analytical flexibility and operational responsiveness.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain the characteristics and architectural implications of high-volume, high-velocity data.

2. Compare distributed storage and computation models.

3. Design batch, streaming, and hybrid data pipelines.

4. Distinguish event sourcing from CDC and select an appropriate approach.

5. Explain event time, processing time, watermarks, windows, and delivery semantics.

6. Formulate data contracts and assess their governance implications.

7. Evaluate architectures using latency, throughput, correctness, cost, and operational risk.

8. Analyse a real-time data-engineering case and justify design decisions.

1. Big Data as an Engineering Problem

“Big data” is not defined by one universal volume threshold. It is better understood as a combination of workload properties. Common dimensions include:

• Volume: the amount of data stored or processed.
• Velocity: the rate at which data is produced, transferred, or required for decisions.
• Variety: the range of structures, formats, and meanings.
• Veracity: uncertainty, incompleteness, inconsistency, and possible bias.
• Value: the business or scientific benefit obtained from the data.

These dimensions create distinct engineering problems. Large volume motivates partitioning and parallel processing. High velocity motivates append-oriented logs, incremental computation, and back-pressure mechanisms. Variety motivates schema management and semantic modelling. Veracity requires validation, reconciliation, and quality monitoring.

A useful performance model begins with three rates:

• Ingress rate is the rate at which events arrive.
• Processing rate is the rate at which a consumer can process events.
• Egress or serving rate is the rate at which results are written or queried.

If a sustained ingress rate exceeds the sustainable processing rate, backlog grows. For a simplified system with constant rates, backlog growth is approximately the difference between ingress and processing rates. In real systems, rates vary, partitions may be imbalanced, and processing may involve external services; therefore capacity planning must include headroom, failure recovery, and peak rather than only average demand.

Latency must also be defined precisely. End-to-end latency is the elapsed time from event creation to availability of a usable result. It includes production, transport, queueing, processing, storage, and serving delays. “Real time” is therefore a service-level objective, not a universal technical category. A fraud decision may require sub-second latency, whereas operational reporting may accept several minutes.

2. Distributed Storage and Computation

2.1 Partitioning and replication

Distributed systems divide data across nodes. A partition is a logical subset of records, commonly selected using a hash key, range, or time interval. Partitioning enables parallel reads and writes, but its effectiveness depends on balanced distribution. A poor key can create a hot partition that receives disproportionate traffic.

Replication maintains multiple copies of data. It improves availability and read capacity, but increases storage and network cost. Replication does not automatically solve correctness problems: replicas must be coordinated, and applications must understand whether reads may be stale.

Distributed storage commonly appears in three forms:

1. Distributed file or object storage: inexpensive, durable storage for large files and table data. It is well suited to data lakes and analytical processing.

2. Distributed databases: systems that provide structured access, indexes, transactions, and often horizontal scaling.

3. Distributed logs: ordered, append-oriented streams of records retained for a configured period. They support replay and decoupling between producers and consumers.

Object storage is often organised as immutable files, with metadata catalogues describing schemas, partitions, and statistics. Table formats built over object storage add capabilities such as snapshots, atomic commits, schema evolution, and record-level updates. Engineers must distinguish physical storage from logical tables: a table is a governed abstraction, not necessarily one file or one database object.

2.2 Consistency, availability, and failure

A distributed system can experience network partitions, node failures, delayed messages, and partial writes. Design choices therefore involve trade-offs among consistency, availability, and partition tolerance. Since network partitions cannot simply be excluded in a distributed deployment, practical systems choose how much availability or consistency to preserve during such failures.

Strong consistency makes the latest committed value visible according to a defined ordering, but may require coordination and incur latency. Eventual consistency permits replicas to converge after updates, improving availability and scalability at the cost of temporary disagreement. Neither is universally superior. A financial ledger generally needs stronger guarantees than a recommendation cache.

A pipeline should specify its failure model. For example, it may tolerate duplicate events but not lost events; it may accept delayed dashboard values but not incorrect account balances. These requirements determine storage, acknowledgement, transaction, and recovery mechanisms.

2.3 Batch computation

Batch engines divide a bounded dataset into tasks, execute transformations in parallel, and combine results. Typical operations include filtering, projection, joins, grouping, and aggregation. Distributed joins can be expensive because records with the same join key must often be brought together. A small reference relation may be replicated to workers, whereas two large relations may require repartitioning or a shuffle.

Batch pipelines are attractive because they are reproducible, efficient for large scans, and relatively easy to validate. They are less suitable when users require continuously updated results or when recomputing the entire dataset is too slow or expensive.

2.4 Data lakes, warehouses, and lakehouses

A data warehouse traditionally stores curated, structured data optimised for analytical queries. A data lake stores data in varied formats, often at lower cost and with greater schema flexibility. A lakehouse seeks to combine low-cost open storage with table management, governance, transactions, and warehouse-like performance.

These labels describe architectural tendencies rather than strict categories. A sound design focuses on workload requirements: ingestion rate, query patterns, retention, update frequency, governance, isolation, and recovery. A single platform may contain an operational database, event log, object store, analytical warehouse, and serving store.

3. Batch, Streaming, and Hybrid Pipelines

A pipeline is a sequence of stages that acquires, transports, validates, transforms, stores, and serves data. A robust pipeline separates concerns while preserving metadata such as source, event time, schema version, processing status, and lineage.

3.1 Batch pipeline pattern

A typical batch pipeline performs the following:

1. Extract data from operational systems or files.

2. Land the raw data in durable storage without destructive modification.

3. Validate structure, required fields, and basic quality rules.

4. Transform and standardise data into curated tables.

5. Publish analytical models or aggregates.

6. Record audit statistics and make the run observable.

The raw layer supports replay and forensic analysis. The curated layer provides stable, documented data for users. A batch job should be idempotent where possible: rerunning the same input should not create unintended additional effects.

3.2 Streaming pipeline pattern

A streaming pipeline generally includes producers, an ingestion layer, a durable event log or broker, stream processors, state stores, sinks, and monitoring. Producers publish records to named streams or topics. Partitions allow parallel consumption while preserving order within a partition. Consumers track their progress so that interrupted work can resume.

Stream processors may perform stateless operations, such as filtering, or stateful operations, such as joins and time-windowed counts. Stateful processing requires durable state, checkpointing, and a recovery strategy. External side effects, such as sending a payment instruction, require special care because retrying a message can repeat the effect.

3.3 Lambda and Kappa perspectives

The lambda architecture uses separate batch and speed paths, then reconciles their outputs. It can provide low latency and correction through recomputation, but duplicates logic and may produce inconsistent results between paths.

The kappa architecture treats the event stream as the primary source and reprocesses retained events when logic changes. It simplifies conceptual flow but requires sufficient event retention, replay capacity, deterministic processing, and effective management of state.

Modern designs often use a hybrid approach: a durable event log, incremental streaming computation, periodic compaction or backfills, and table formats that support both historical and near-real-time access. The important principle is to make the source of truth and reconciliation strategy explicit.

4. Event Sourcing and Change Data Capture

4.1 Event sourcing

In event sourcing, changes to an entity are represented as an ordered sequence of domain events rather than only as the latest mutable row. For an account, events might include `AccountOpened`, `DepositRecorded`, and `WithdrawalRecorded`. The current state is derived by replaying events, possibly with periodic snapshots.

Event sourcing provides an audit trail, temporal reconstruction, and the ability to create new projections from historical events. It also imposes design obligations. Events must have stable meaning, identifiers, ordering rules, versioning policies, and explicit handling of corrections. An event is a business fact, not merely a database update.

4.2 Change data capture

CDC records changes made to an existing source database. It may use transaction logs, triggers, timestamps, or database-specific replication mechanisms. A CDC record commonly includes the operation type, key, before image, after image, transaction metadata, and source position.

CDC is valuable for integrating legacy systems without changing application behaviour, replicating data, building analytical copies, and feeding search or cache systems. However, CDC does not necessarily reveal the business reason for a change. A changed status field may result from several different business actions. It also inherits source-database limitations and may expose sensitive fields.

4.3 Comparing the approaches

Event sourcing records intentional domain events at the point of business action. CDC records persistence changes at the database boundary. Event sourcing is semantically rich but requires an event-oriented application design. CDC is often easier to introduce into existing systems but may be semantically weaker.

They can be combined. A service may use domain events for business integration and CDC for synchronising operational tables. Architects should avoid treating a technical row update as automatically equivalent to a meaningful business event.

5. Stream-Processing Semantics

5.1 Event time and processing time

Event time is when an event occurred in the domain, such as when a sensor measurement was taken. Processing time is when a system processes the record. The two differ because of network delay, retries, batching, device disconnection, or clock error.

Applications that report activity by occurrence usually need event-time processing. Processing-time windows are simpler but can misclassify delayed records. A system should state which time is used and how timestamps are validated.

5.2 Windows and watermarks

A window groups events for computation. Tumbling windows are fixed and non-overlapping; sliding windows overlap; session windows group activity separated by less than a defined inactivity gap.

A watermark is an estimate that the system has progressed sufficiently far in event time to evaluate a window. It is not proof that no later event will arrive. A design therefore needs a policy for late data: discard it, update the result, route it for correction, or maintain an uncertainty interval.

For example, a five-minute event-time window with a two-minute allowed lateness may be finalised only after the watermark passes the window end plus two minutes. This improves completeness but increases result latency. The choice is a business trade-off between timeliness and correction.

5.3 Delivery and processing guarantees

Common delivery descriptions are:

• At-most-once: a record may be lost, but the system does not intentionally retry it.
• At-least-once: a record is retried until acknowledged, so duplicates are possible.
• Effectively-once or exactly-once processing: the combination of computation and state/output management is arranged so that each logical input contributes once to the intended result, subject to stated system boundaries.

“Exactly once” must not be used casually. A processor may atomically update internal state and a managed sink, yet an external email service may still receive duplicate requests. Idempotency keys, deduplication, transactional sinks, and reconciliation are often necessary.

An idempotent operation produces the same final effect when applied repeatedly with the same logical request. For example, setting an order status to `SHIPPED` using an order identifier can be made idempotent; blindly appending a charge is not.

6. Data Contracts and Schema Evolution

A data contract is an explicit agreement between a producer and its consumers. It should define field names and types, requiredness, semantic meaning, units, valid ranges, ownership, privacy classification, event identity, timestamps, retention, compatibility expectations, and service-level objectives.

A schema is not the whole contract. A field called `amount` is incomplete without currency, precision, sign convention, and whether it represents an authorised or settled amount. Contracts should therefore include semantic and operational requirements.

Schema evolution may be backward compatible, forward compatible, both, or neither. Adding an optional field is commonly safer than renaming or changing the meaning of an existing field. Compatibility must be tested against real consumers, not inferred only from a schema registry. Breaking changes should use a new version, migration period, or translation layer.

Data quality should be observable. Useful indicators include completeness, validity, uniqueness, timeliness, consistency, and distribution drift. A quality failure should have an ownership and incident process. Silent coercion is dangerous: converting an invalid value to zero can create plausible but incorrect analytics.

7. Designing for Scale and Reliability

7.1 Capacity and parallelism

Partition count should reflect expected throughput, consumer parallelism, storage layout, and future growth. Too few partitions limit parallelism; too many increase metadata and coordination overhead. Partition keys should distribute load while preserving any ordering required by the business. Global ordering is expensive and usually unnecessary; ordering per entity or account is often sufficient.

Back pressure occurs when downstream processing cannot keep up. Systems may buffer, throttle producers, shed non-critical work, scale consumers, or degrade gracefully. Unbounded buffering merely converts a throughput problem into an eventual failure.

7.2 Reliability patterns

Important patterns include:

• durable raw ingestion and replayable logs;
• checkpoints and committed offsets;
• dead-letter or quarantine streams for invalid records;
• retry policies with exponential backoff and limits;
• idempotent writes and deduplication keys;
• circuit breakers around unstable dependencies;
• schema and data-quality validation at boundaries;
• disaster recovery with tested restoration procedures;
• observability covering logs, metrics, traces, lag, throughput, freshness, and error rates.

A runbook should explain how operators identify a stuck partition, replay a time range, correct a bad deployment, and reconcile outputs.

7.3 Cost and sustainability

Costs arise from storage, network transfer, compute, replication, managed services, and operational labour. High-frequency small files can damage analytical performance by increasing metadata overhead. Compaction, appropriate file sizing, partition pruning, retention policies, and tiered storage are practical controls.

Sustainable design avoids processing unchanged data, selects fit-for-purpose retention, and measures energy-intensive workloads where possible. Lower cost is not a success if it undermines recovery or data quality.

8. Security, Governance, and Ethics

Streaming systems can move sensitive data at high speed, making errors difficult to contain. Security controls should include identity-based authentication, least-privilege authorisation, encryption in transit and at rest, secret management, network segmentation, audit logging, and controlled administrative access.

Privacy governance requires purpose limitation, data minimisation, retention limits, access review, and appropriate deletion or suppression mechanisms. Immutable logs create a tension with deletion obligations. A practical design may separate identifying information from event facts, encrypt or tokenise identifiers, apply retention policies, and maintain deletion-aware projections. Legal requirements vary by jurisdiction and sector, so technical design must be aligned with organisational legal advice.

Data lineage and provenance support accountability: an analyst should be able to identify the source, transformations, versions, and owners of a reported value. Automated decisions based on streaming data require attention to bias, explainability, contestability, and disparate impact. A low-latency model can amplify an erroneous or biased signal at scale.

9. Applied Case: Illustrative Real-Time Payment Risk Platform

Background

Consider a fictional national payment provider processing card authorisations, merchant events, customer-device signals, and historical transaction data. The provider wants to reduce fraudulent approvals while keeping legitimate authorisations below a defined latency target.

Problem

The existing system creates nightly reports. Fraud analysts receive delayed information, and a mutable operational database cannot support high-volume analytical queries without affecting authorisation performance. Events may arrive late from mobile devices, and duplicate delivery can occur during network recovery.

Analysis

The system requires an immediate decision path and a historical analytical path. The key entity for ordering is the transaction identifier, while device and customer aggregates require state across multiple events. The provider must not assume that a message is processed only once. A fraud decision is also a consequential decision, so reason codes, auditability, access control, and model monitoring are required.

Approach and solution

The provider introduces the following architecture:

1. The authorisation service emits a versioned transaction event containing an event identifier, event time, transaction time, merchant category, amount and currency, tokenised customer reference, and source metadata.

2. A durable partitioned log retains events long enough to support replay and investigation.

3. A streaming processor validates the contract, rejects malformed records to a quarantine stream, and maintains event-time aggregates such as recent transaction counts by token and device.

4. A low-latency feature store serves current features to a risk model. The model returns a decision and reason codes; the original event and decision are retained for audit.

5. A warehouse or lakehouse receives the event stream and periodic snapshots for analyst queries and model training.

6. A reconciliation job compares authorisation outcomes with settlement and chargeback data. Corrections are represented as new events or controlled updates rather than silent overwrites.

7. Monitoring tracks consumer lag, event freshness, rejected-record rate, feature staleness, model drift, decision distribution, and false-positive indicators.

At-least-once delivery is accepted, while event identifiers and idempotent writes prevent duplicate logical transactions. Late events update analytical aggregates according to a documented correction policy, but the authorisation decision is not retroactively changed. This distinction separates immediate operational action from later analytical truth.

Outcome

The provider obtains rapid risk decisions without running heavy queries on the transactional database. Analysts can replay events, investigate decisions, and rebuild derived features. The platform still requires governance: model performance must be assessed across relevant populations, access to payment data must be restricted, and retention must match legal and business requirements.

Learning points

The case demonstrates that low latency alone is insufficient. Correct identity, event time, replay, idempotency, contracts, lineage, and human governance are equally important. It also shows why the operational decision path and the analytical correction path should have explicit boundaries.

10. Contemporary Developments and Critical Evaluation

Several developments are shaping data engineering. Open table formats and interoperable catalogues seek to reduce dependence on proprietary storage layers. Streaming SQL and unified batch-stream engines make incremental computation more accessible. Data mesh approaches emphasise domain ownership and data products, but they do not eliminate the need for platform standards, contracts, and central governance. Data fabric initiatives focus on metadata, integration, and policy enforcement across heterogeneous environments.

Artificial intelligence is relevant in two ways. First, machine-learning models consume streaming features and produce real-time predictions. Second, AI-assisted tools can help generate pipeline code, tests, documentation, and anomaly alerts. Such tools do not remove the need for human review: generated transformations may change semantics, expose confidential information, or fail under rare failure modes. Model-serving pipelines additionally require feature consistency, drift monitoring, reproducibility, and responsible-decision controls.

A critical architectural question is whether every workload needs streaming. Streaming introduces state management, operational complexity, late-data policy, and cost. If a daily refresh meets the decision requirement, batch may be the more reliable choice. Conversely, forcing a high-velocity workload into periodic batch processing can create unacceptable risk. Architecture should follow measurable service objectives rather than fashionable terminology.

Best-Practice Checklist

• Define freshness, latency, completeness, and correctness objectives separately.
• Identify the source of truth and the replay or reconciliation strategy.
• Use stable event identifiers and make side effects idempotent.
• Partition for balanced load and required ordering, not convenience alone.
• Treat event time, late data, and clock quality as explicit design concerns.
• Publish data contracts with semantic, privacy, and operational requirements.
• Keep raw data sufficiently durable for recovery and controlled reprocessing.
• Monitor lag, freshness, quality, cost, and business outcomes.
• Test failure recovery, schema evolution, replay, and back-pressure behaviour.
• Apply privacy, security, lineage, and responsible-AI controls from the design stage.

Chapter Summary

Big data and real-time data engineering combine distributed storage, parallel computation, durable event transport, and governed data products. Batch processing is effective for bounded, reproducible computation; streaming is effective for continuous, low-latency decisions. Hybrid architectures can combine both when their source of truth and reconciliation rules are explicit.

Partitioning enables scale, replication improves resilience, and distributed logs support decoupling and replay. Event sourcing captures domain facts, whereas CDC captures changes at a persistence boundary; they solve related but different problems. Event-time processing, windows, watermarks, late-data policies, idempotency, and delivery semantics determine whether streaming results are timely and trustworthy. Data contracts connect technical schemas with business meaning, ownership, quality, privacy, and compatibility.

Successful platforms optimise more than throughput. They address failure recovery, observability, security, cost, sustainability, regulatory obligations, and the social consequences of automated decisions. The appropriate architecture is the simplest one that satisfies measurable requirements for latency, correctness, availability, scale, and governance.

Key Terms

Big data: Data whose scale, speed, diversity, or uncertainty creates specialised engineering challenges.

Batch processing: Processing a bounded dataset, usually on a schedule.

Stream processing: Continuous processing of an unbounded sequence of events.

Partition: A logical subdivision of distributed data used for parallelism and scaling.

Replication: Maintaining multiple copies of data for resilience or read capacity.

Distributed log: An append-oriented, retained sequence of records supporting ordered consumption and replay.

Event time: The time at which an event occurred in the domain.

Processing time: The time at which a system processes an event.

Watermark: An estimate of progress used to decide when event-time results can be evaluated.

Window: A defined grouping of events for temporal computation.

Late data: An event received after the system’s expected event-time progress.

Event sourcing: Persisting domain changes as an ordered sequence of business events.

Change data capture: Recording inserts, updates, and deletes from a source system.

Data contract: An explicit agreement covering a data product’s schema, meaning, quality, ownership, and operational expectations.

Idempotency: The property that repeating the same logical operation does not create an unintended additional effect.

Back pressure: A mechanism by which downstream capacity constraints regulate upstream production.

At-least-once delivery: Delivery in which retries prevent intentional loss but duplicates may occur.

Data lakehouse: An architecture combining open, low-cost storage with managed tables and analytical capabilities.

Data lineage: Traceability of data from its origin through transformations to its uses.

Review Questions

Short Answer Questions

1. Distinguish event time from processing time.

2. What is the purpose of a watermark?

3. State one important difference between event sourcing and CDC.

4. Why are partition keys important in a distributed log?

5. Define idempotency in the context of data pipelines.

Long Answer Questions

1. Compare batch, streaming, and hybrid architectures, including their benefits and limitations.

2. Explain how partitioning, replication, and consistency choices affect distributed data systems.

3. Discuss delivery semantics and explain why “exactly once” requires qualification.

4. Describe the contents and purpose of a comprehensive data contract.

5. Analyse the governance, privacy, and security challenges of real-time data platforms.

Analytical / Application Questions

1. A stream receives events at a sustained rate higher than its processing capacity. Analyse the likely consequences and propose a capacity and back-pressure strategy.

2. Design an event-time windowing policy for a sensor system in which devices can be offline for several minutes. Justify the watermark, lateness, correction, and serving choices.

3. For an established retail database, evaluate whether event sourcing, CDC, or a combination should feed a real-time analytics platform. State assumptions and defend your recommendation.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly motivates partitioned parallel processing?

A. High volume

B. Low data diversity

C. Strong user authentication

D. Short field names

Correct Answer: A. High volume

Question 2

What does a watermark primarily represent in event-time processing?

A. The exact time of the last event that will ever arrive

B. An estimate of progress used to evaluate time-based results

C. The physical location of a data replica

D. The encryption status of a message

Correct Answer: B. An estimate of progress used to evaluate time-based results

Question 3

Which statement best distinguishes CDC from event sourcing?

A. CDC records database changes, while event sourcing records domain events

B. CDC always provides richer business meaning than event sourcing

C. Event sourcing can only be used for analytical databases

D. Event sourcing cannot support replay

Correct Answer: A. CDC records database changes, while event sourcing records domain events

Question 4

Which delivery guarantee permits duplicates because messages may be retried?

A. At-most-once

B. At-least-once

C. No-delivery

D. Single-reader delivery

Correct Answer: B. At-least-once

Question 5

Which property is most useful when a retried pipeline operation must not create an additional charge?

A. Idempotency

B. Compression

C. Partition pruning

D. Eventual deletion

Correct Answer: A. Idempotency

Question 6

Why is global ordering usually avoided in high-scale event systems?

A. It prevents all data validation

B. It is expensive and often unnecessary for business processing

C. It makes encryption impossible

D. It eliminates replay

Correct Answer: B. It is expensive and often unnecessary for business processing

Question 7

Which item belongs in a data contract beyond a field’s physical type?

A. The developer’s preferred programming language

B. The field’s semantic meaning and unit

C. The colour of the dashboard

D. The number of office locations

Correct Answer: B. The field’s semantic meaning and unit

Question 8

A system processes a five-minute event-time window and permits two minutes of lateness. Relative to the window end, when should finalisation normally be considered?

A. Immediately at the window start

B. At the window end, without qualification

C. After the watermark passes the window end plus the allowed lateness

D. Only after permanent deletion of the source data

Correct Answer: C. After the watermark passes the window end plus the allowed lateness

Question 9

Which architecture uses separate batch and speed paths that must be reconciled?

A. Lambda architecture

B. Single-table architecture

C. Peer-to-peer cache architecture

D. Static file architecture

Correct Answer: A. Lambda architecture

Question 10

Which is a principal risk of an imbalanced partition key?

A. Hot partitions and reduced parallel performance

B. Automatic elimination of duplicates

C. Guaranteed stronger consistency

D. Removal of all network traffic

Correct Answer: A. Hot partitions and reduced parallel performance