Streaming and Real-Time Data Engineering

Introduction

Traditional data engineering assumes that data can be collected, stored, and processed in batches. This model remains useful for financial close, periodic reporting, and large-scale historical analysis. However, many business decisions depend on what is happening now: a payment may need authorization within milliseconds, a delivery platform may need to reposition drivers continuously, and a security system may need to identify suspicious behavior before an attack progresses.

Streaming and real-time data engineering addresses this need by treating data as a continuously arriving sequence of events. Rather than waiting for a dataset to become complete, a streaming application processes records incrementally. The central challenge is not merely speed. A reliable streaming system must also handle disorder, duplication, failure, changing schemas, late records, state recovery, and the difference between when an event occurred and when it was processed.

This chapter develops the conceptual and technical foundations of event-driven pipelines. It examines messaging systems, publish-subscribe architectures, event logs, windowing, stateful processing, watermarking, delivery semantics, stream-table duality, and real-time analytics patterns. The discussion is oriented toward graduate students who must evaluate both engineering design and business consequences.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish event time, processing time, ingestion time, latency, and throughput.
2. Explain the roles of producers, brokers, topics, partitions, consumers, and consumer groups.
3. Design publish-subscribe pipelines using durable event logs.
4. Select appropriate windows and reason about late and out-of-order events.
5. Explain stateful stream processing, checkpoints, and recovery.
6. Compare at-most-once, at-least-once, and exactly-once processing semantics.
7. Apply stream-table duality to materialized views and real-time applications.
8. Evaluate operational, governance, security, and ethical risks in streaming systems.

1. From Batch Data to Event-Driven Data

An event is a record that describes something that happened. Examples include OrderPlaced, PaymentAuthorized, SensorReadingReceived, and CustomerProfileUpdated. An event generally contains a payload, a timestamp, an identifier, and metadata such as source, schema version, and correlation identifier.

A useful conceptual model is:

Source systems → Event producers → Broker or event log → Stream processors → Sinks and applications

A producer publishes events without needing to know every downstream consumer. The broker temporarily or durably stores events and makes them available to consumers. A processor transforms, filters, joins, aggregates, or enriches the stream. Sinks may include operational databases, analytical warehouses, dashboards, alerting systems, or machine-learning services.

1.1 Events, commands, and state changes

An event records a fact: “Order 481 was shipped.” A command requests an action: “Ship order 481.” Confusing commands and events can produce unsafe designs. Commands are directed to a responsible service and may be rejected; events are observations that can be consumed by multiple parties.

Events may be immutable facts, while current state is often derived from them. For example, an account balance can be represented as a current value in a database or reconstructed from account-opening, deposit, withdrawal, and correction events. Event-based reconstruction improves auditability but requires careful ordering, correction, and retention policies.

1.2 Time and real-time performance

Streaming systems use several notions of time:

• Event time: when the business event occurred at its source.
• Ingestion time: when a platform accepted the event.
• Processing time: when a processor handled the event.
• Publication time: when the event became available to consumers.

These values can differ because of network delay, offline devices, retries, and queue backlogs. A dashboard that uses processing time may show a different result from one based on event time.

Latency is commonly measured as elapsed time between two points. If an event occurs at time t_e and an alert is emitted at t_a, end-to-end alert latency is:

Here, L is latency, and all times must use the same clock basis and units. In distributed systems, clock synchronization and timestamp quality are therefore operational concerns. Throughput is the number of records or bytes processed per unit of time, whereas latency concerns the delay experienced by an individual event. Increasing throughput does not automatically reduce latency.

2. Messaging Systems and Event Logs

2.1 Core broker concepts

A message broker mediates communication between producers and consumers. Important components include:

In a partitioned topic, ordering is normally guaranteed within a partition, not globally across all partitions. A partition key, such as customer_id or account_id, can preserve ordering for a business entity while allowing parallel processing across entities.

2.2 Queues and publish-subscribe

In a queue model, a message is generally delivered to one member of a worker group. This is suitable for distributing tasks such as image processing. In a publish-subscribe model, multiple independent subscriber groups can receive the same event. This is useful when an OrderPlaced event must be consumed by inventory, billing, recommendation, and analytics services.

A durable event log combines publish-subscribe distribution with retention. Consumers can read at different speeds, restart from a prior offset, and replay historical events. Replay supports debugging, rebuilding materialized views, and recovering downstream systems. It does not eliminate the need for data-quality controls: replaying a flawed event stream reproduces the flaw.

2.3 Partitioning and scalability

Partitioning provides parallelism. If a topic has P partitions and a consumer group has C active consumers, useful parallelism is bounded approximately by:

C_eff is the maximum number of consumers that can actively own distinct partitions at one time, assuming one consumer can process its assigned workload and that partition workloads are balanced. More consumers than partitions do not increase partition-level parallelism. More partitions can improve throughput but may increase coordination, storage, and operational complexity.

Poor key selection can create a hot partition. For example, using a single region as the key may send most events to one partition. Random keys distribute load but may destroy per-customer ordering. Key design is therefore a business and technical decision.

3. Delivery Semantics and Reliability

A streaming pipeline must define what happens when failures occur between reading an event, performing computation, and writing an output.

• At-most-once: an event is processed zero or one time. Loss is possible, but duplicates are avoided.
• At-least-once: an event is processed one or more times. Loss is minimized, but duplicates are possible.
• Exactly-once effect: the externally visible result is equivalent to processing each logical event once, usually through transactions, idempotent writes, deduplication, and coordinated state management.

“Exactly once” should not be treated as a universal property. A processor may provide exactly-once state updates while an external email service still sends duplicate emails after a timeout. End-to-end guarantees depend on every boundary.

An idempotent operation can be safely repeated. If a payment status update sets transaction 92 to AUTHORIZED, repeating the same update may be harmless. An operation that adds $10 to a balance is not idempotent unless it is protected by a unique transaction identifier. A common deduplication pattern stores processed event IDs and rejects IDs already committed.

Reliability also involves checkpointing. A checkpoint records sufficient progress and state to resume after failure. If a processor commits its input offset before its output is durable, a failure may cause data loss. If it commits output before its offset, a restart may produce duplicates. Atomic transactions or idempotent sinks help resolve this coordination problem.

4. Stateful Stream Processing

Stateless transformations process each event independently. Filtering invalid records or converting temperature from Celsius to Fahrenheit are examples. Stateful transformations depend on prior events. Examples include counting purchases, maintaining a running balance, detecting a sequence of logins, and joining transactions to customer profiles.

A stateful operator can be described conceptually as:

Here, S_t is operator state before event e_t, F is the update function, and S_t+1 is the new state. The expression is conceptual rather than a guarantee that events arrive in perfect order. Real systems must define how late, duplicate, and corrected events affect state.

State may be held in memory for speed, on local durable storage, or in an external state store. Large state requires memory management, compaction, checkpointing, and recovery planning. State retention should be bounded whenever business requirements permit it. An unbounded distinct-user set, for example, can eventually exhaust resources.

4.1 Windows

Because streams are theoretically unbounded, many calculations use finite windows.

• Tumbling window: fixed, non-overlapping intervals, such as each five-minute period.
• Hopping window: fixed-length intervals that overlap, such as a 10-minute window evaluated every minute.
• Sliding window: continuously updated according to time or count.
• Session window: dynamically groups activity separated by an inactivity gap.
• Count window: contains a specified number of events rather than a time duration.

For a tumbling window of width w, event time t belongs to a window determined by an agreed boundary rule, often an interval of the form [kw, (k+1)w), where k is an integer. Boundary conventions matter: an event exactly at a boundary must be assigned consistently.

A five-minute average is not necessarily the average of five one-minute averages. The correct aggregate depends on the underlying measure and weights. If one-minute groups have different counts, an overall mean must be weighted by observations:

Here, x̄_i is subgroup mean and m_i is subgroup count. This distinction is important in streaming dashboards.

4.2 Watermarks and late data

A watermark is an operational estimate that the system has probably observed events up to a particular event-time frontier. It is not proof that no earlier event will ever arrive. If the watermark is W, the system may close or finalize windows whose end is sufficiently earlier than W, subject to the platform’s lateness policy.

A practical policy may specify:

1. how long to wait for out-of-order events;
2. whether late events update a previously emitted result;
3. whether corrections are emitted as updates or retractions; and
4. how much state is retained after window closure.

Waiting longer improves completeness but increases result latency and state cost. Waiting less produces faster but potentially incomplete results. This is a business trade-off, not merely a technical parameter.

5. Stream-Table Duality and Joins

A stream is a sequence of changes or events. A table is a view of current or historical state. The stream-table duality states that a table can be represented as a changelog stream, while a stream can be materialized into a table by applying its records in order.

For example, customer status events may produce the table:

A new status event changes the materialized table. The table provides efficient lookup; the stream provides history and reactivity.

This duality supports a common pattern: joining a high-volume transaction stream with a relatively stable customer table. The processor may maintain a local, continuously updated copy of customer attributes. If the customer table changes, the system must decide whether past transactions should be reinterpreted or whether the join should use the attributes valid at transaction time. The latter is a temporal join and is essential for accurate historical analysis.

A stream-stream join generally requires windows because both inputs are unbounded. A stream-table join can often use a current-state lookup, but consistency and update timing must be specified. Late dimension updates can change results unless historical versions are retained.

6. Real-Time Analytics Patterns

6.1 Threshold alerts

A processor compares each event or aggregate with a threshold. Examples include inventory below reorder level, temperature above a safety limit, or transaction value above a risk boundary. Threshold alerts are simple but can generate alert fatigue. Hysteresis, suppression intervals, and severity levels can prevent repeated notifications.

6.2 Rolling metrics and anomaly detection

A rolling metric summarizes recent behavior. A moving average over the last n observations is:

Here, x_t is the current observation and n is the window size. The formula assumes observations are equally weighted and regularly indexed; for irregular event times, a time-based or weighted method may be more appropriate.

An exponentially weighted moving average can react more quickly to recent changes:

α controls responsiveness. Larger values emphasize the newest observation. The initial value E₀ must be specified, and the method is not a complete anomaly-detection system by itself.

6.3 Complex event processing

Complex event processing detects patterns across multiple events, such as three failed logins followed by a successful login from a new device. Pattern systems must define time bounds, correlation keys, partial matches, and cancellation conditions. A broad pattern may detect more threats but produce more false positives.

6.4 Change data capture

Change data capture, or CDC, converts database inserts, updates, and deletes into events. CDC enables downstream systems to react without repeatedly scanning an entire source table. Reliable CDC requires transaction ordering, primary keys, schema evolution, delete handling, and a strategy for initial snapshots.

6.5 Lambda and streaming-first architectures

A traditional lambda architecture separates batch and speed layers, potentially creating two implementations of business logic. A streaming-first architecture uses one continuously updated pipeline and periodically corrects or recomputes results when necessary. The best choice depends on correctness requirements, historical replay capability, cost, and operational maturity.

7. Illustrative Case: Real-Time Payment Risk Screening

This is a fictional illustrative case. MeridianPay, a digital payments provider, processes card-not-present transactions. Its existing batch fraud report runs every hour, allowing suspicious transactions to remain active too long. The company wants risk decisions within two seconds while preserving an auditable history.

Background and problem

Each authorization generates an event containing transaction ID, customer ID, merchant ID, amount, currency, device fingerprint, event timestamp, and location. Events may be duplicated by retries and may arrive out of order. The system must not block every legitimate customer when a risk service is unavailable.

Analysis

The team identifies three classes of features:

1. per-transaction features, such as amount and device novelty;
2. short-window features, such as transaction count and total value for a customer during the previous 10 minutes; and
3. reference features, such as customer risk tier and merchant category.

The short-window features require keyed state and event-time windows. The reference features are maintained as a materialized customer table. A transaction identifier provides an idempotency key.

Approach

The architecture uses a durable authorization topic partitioned by customer ID. A risk processor consumes the topic, validates the schema, rejects impossible amounts, and updates keyed state. A watermark allows modest out-of-order arrival, while a late-event policy emits corrections for analytical outputs but does not retroactively change an already settled authorization without a separate review process.

The processor writes a decision record to an idempotent decision store. It also publishes RiskDecisionMade for monitoring and case management. The decision store uses transaction ID as a unique key, so retries do not create multiple decisions. Sensitive fields are encrypted in transit and at rest, and access to raw payment events is restricted.

Outcome

The new design reduces decision latency and makes fraud features available continuously. It also exposes trade-offs: a customer who transacts while offline may generate late events; a strict two-second service objective limits how long the system can wait for reference-data updates; and model changes require versioned features and reproducible decisions.

Learning points

The case demonstrates that real-time analytics is a socio-technical system. Low latency alone is insufficient. Correct partitioning, state management, idempotency, temporal assumptions, model governance, and fallback behavior determine whether the system is trustworthy.

8. Operations, Governance, and Ethical Issues

Streaming systems require observability across the entire path. Important metrics include consumer lag, processing latency, end-to-end latency, throughput, error rate, retry volume, late-event rate, state size, checkpoint duration, and dead-letter volume. Lag should be interpreted carefully: a consumer may have low record lag but high event-time delay if producers are delayed.

Schema governance is essential. Backward-compatible changes, explicit versioning, required fields, data types, and ownership reduce pipeline breakage. A schema registry can enforce compatibility rules, but governance must also define meanings, quality expectations, retention, and permitted uses.

Privacy risks can be amplified by real-time systems because data becomes actionable immediately. Organizations should apply purpose limitation, data minimization, access control, encryption, retention limits, and audit logging. Location, behavioral, and inferred-risk data may require heightened safeguards. Automated decisions should be explainable to the extent required by law and policy, and models should be evaluated for disparate error rates across relevant groups.

Operational failure modes include broker outages, poisoned messages, hot partitions, unbounded state, clock errors, schema incompatibility, and downstream overload. Backpressure, rate limiting, circuit breakers, dead-letter topics, replay procedures, and disaster-recovery exercises are standard defensive practices. A dead-letter topic is not a final solution: teams must monitor it, classify failures, and establish safe reprocessing procedures.

Contemporary developments

Cloud platforms increasingly provide managed brokers, serverless stream processors, lakehouse tables, and integrated feature stores. This reduces infrastructure administration but does not remove architectural responsibility. Costs can rise through excessive retention, cross-region transfer, high-cardinality state, or unnecessary materialized views.

AI and machine learning are increasingly embedded in streaming pipelines for anomaly detection, recommendations, forecasting, and adaptive routing. These systems introduce model-serving latency, concept drift, feature consistency, and feedback-loop risks. A model that influences which events are observed can change its own training population. Therefore, real-time ML requires model versioning, feature lineage, drift monitoring, and human escalation for high-impact decisions.

9. Design Best Practices

1. Define business latency, completeness, and correction requirements before selecting technology.
2. Use event time when business meaning depends on when activity occurred.
3. Choose partition keys that balance load while preserving required ordering.
4. Design every sink for idempotency or explicitly accept duplicate effects.
5. Bound state and document retention assumptions.
6. Treat watermarks as estimates, not guarantees of completeness.
7. Version schemas and test compatibility in deployment pipelines.
8. Separate raw immutable events from derived, replaceable views.
9. Instrument lag, latency, errors, state growth, and late data.
10. Test replay, failover, backpressure, duplicate delivery, and out-of-order events.
11. Protect sensitive data throughout the event lifecycle.
12. Document ownership, service-level objectives, and operational runbooks.

Chapter Summary

Streaming and real-time data engineering models business activity as continuously arriving events. Messaging systems decouple producers and consumers, while durable partitioned event logs provide scalability, replay, and independent subscriber progress. Delivery semantics determine how failures affect loss and duplication; exactly-once effects require coordinated state, transactions, or idempotent external operations.

Stateful processing supports windows, joins, pattern detection, and real-time metrics. Event time, watermarks, and late-data policies are necessary because distributed events do not arrive in perfect order. Stream-table duality explains how event histories produce current-state tables and how materialized tables can serve responsive applications. Real-time analytics patterns include alerts, rolling measures, complex event processing, CDC, and online risk screening.

Successful systems balance latency against completeness, throughput against ordering, and automation against governance. Privacy, explainability, observability, schema management, recovery, and model governance are integral design requirements rather than optional additions.

Key Terms

Event: A record describing something that happened.

Event time: The time at which an event occurred in the business or source context.

Processing time: The time at which a processor handles an event.

Producer: A component that publishes events.

Consumer: A component that reads and processes events.

Topic: A logical category or stream of related events.

Partition: An ordered subdivision of a topic used for storage and parallelism.

Consumer group: Consumers cooperating to process partitions of a topic.

Offset: A position identifying progress through a partition.

Event log: A durable, append-oriented record of events.

Idempotency: The property that repeating an operation does not change its intended final effect.

At-least-once delivery: A guarantee that permits duplicate processing but aims to avoid loss.

Window: A bounded portion of a stream used for aggregation or pattern detection.

Watermark: An estimate of progress through event time.

Late event: An event arriving after the system’s expected event-time frontier.

Stateful processing: Processing in which results depend on retained prior information.

Stream-table duality: The relationship in which streams represent changes and tables represent derived state.

Change data capture: Conversion of database changes into an event stream.

Backpressure: A mechanism by which downstream limits slow upstream production or processing.

Materialized view: A maintained, queryable result derived from source data or events.

Review Questions

Short Answer Questions

1. Distinguish event time from processing time.
2. Why is ordering normally guaranteed only within a partition?
3. Define at-least-once delivery and identify its main risk.
4. What problem does a watermark address?
5. Explain the meaning of stream-table duality.

Long Answer Questions

1. Compare queue-based messaging, publish-subscribe systems, and durable event logs.
2. Explain how partition-key selection affects scalability, ordering, and hot-partition risk.
3. Discuss stateful stream processing, checkpointing, and recovery after failure.
4. Compare tumbling, hopping, sliding, session, and count windows, including suitable applications.
5. Evaluate the governance and privacy requirements of a real-time customer analytics platform.

Analytical / Application Questions

1. A retailer receives duplicate and out-of-order purchase events. Design a pipeline that computes customer spending over 15-minute event-time windows. Specify partition key, deduplication method, watermark policy, and handling of late events.
2. A streaming service has 12 partitions and 20 consumers in one consumer group. What is the maximum partition-level consumer parallelism, and what operational issue may arise from the six unassigned consumers? Explain your reasoning.
3. Design a real-time fraud-screening architecture that combines a transaction stream with changing customer attributes. Explain how you would manage temporal consistency, idempotent decisions, model versioning, privacy, and failure recovery.

Multiple Choice Questions with Answers

Question 1

Which statement best describes event time?

A. The time at which a consumer reads an event

B. The time at which an event occurred in its source or business context

C. The time at which a broker deletes an event

D. The time at which a dashboard refreshes

Correct Answer: B. The time at which an event occurred in its source or business context

Question 2

What is the principal purpose of partitioning a topic?

A. To eliminate the need for schemas

B. To guarantee global ordering across all events

C. To support parallelism while providing ordering within each partition

D. To prevent consumers from replaying events

Correct Answer: C. To support parallelism while providing ordering within each partition

Question 3

Which delivery semantic permits duplicate processing but aims to avoid data loss?

A. At-most-once

B. At-least-once

C. Best-effort only

D. No-delivery

Correct Answer: B. At-least-once

Question 4

Why are watermarks used in event-time processing?

A. To estimate progress and decide when windows may be closed

B. To encrypt messages in transit

C. To assign partitions randomly

D. To replace schema validation

Correct Answer: A. To estimate progress and decide when windows may be closed

Question 5

Which operation is naturally idempotent when implemented as stated?

A. Add $10 to an account balance each time the event is received

B. Send a promotional email for every retry

C. Set transaction 92's status to AUTHORIZED using transaction 92 as a unique key

D. Increment a counter without deduplication

Correct Answer: C. Set transaction 92's status to AUTHORIZED using transaction 92 as a unique key

Question 6

A topic has 12 partitions and a consumer group has 20 active consumers. Ignoring other constraints, how many consumers can actively own partitions at the same time?

A. 8

B. 12

C. 20

D. 32

Correct Answer: B. 12

Question 7

Which window uses fixed, non-overlapping intervals?

A. Tumbling window

B. Hopping window

C. Session window

D. Unbounded window

Correct Answer: A. Tumbling window

Question 8

What does stream-table duality imply?

A. Tables cannot be updated from streams

B. A stream can represent changes used to derive current table state

C. Streams are always smaller than tables

D. Tables guarantee event ordering

Correct Answer: B. A stream can represent changes used to derive current table state

Question 9

Which design most directly addresses duplicate external effects during retries?

A. Removing all checkpoints

B. Using idempotent writes or deduplication keys

C. Increasing the number of dashboard users

D. Ignoring event identifiers

Correct Answer: B. Using idempotent writes or deduplication keys

Question 10

Which metric most directly indicates that consumers are falling behind producers?

A. Consumer lag

B. Schema version

C. Partition key length

D. Encryption algorithm name

Correct Answer: A. Consumer lag