Real-Time Data and Stream Processing

Introduction

Traditional data processing assumes that data can be collected, stored, and processed in batches. This model remains useful for periodic reporting, but many modern systems must react while events are occurring. A payment platform may need to detect fraud before authorizing a transaction; a logistics company may need to reroute vehicles as conditions change; and an online service may need to identify an outage within seconds rather than after the next daily report.

Real-time data processing addresses these requirements by treating continuously generated records as event streams. Instead of waiting for a finite dataset, a streaming application consumes events, maintains relevant state, computes results incrementally, and emits outputs with bounded delay. The central engineering challenge is to achieve an appropriate balance among latency, correctness, scalability, cost, and resilience.

This chapter examines the concepts and technologies used to design streaming systems. It focuses on event streams, brokers, partitions, time semantics, windows, state, watermarks, delivery guarantees, and fault tolerance. Kafka is used as a representative event-broker technology, while Apache Flink and Spark Structured Streaming illustrate stream-processing engines.

Learning Objectives

After completing this chapter, students should be able to:

1. Explain the architecture and terminology of event-streaming systems.
2. Distinguish event time, processing time, and ingestion time.
3. Select appropriate windowing and state-management strategies.
4. Explain watermarks and the treatment of late data.
5. Compare at-most-once, at-least-once, and exactly-once processing.
6. Design a basic streaming solution using Kafka with Flink or Spark Structured Streaming.
7. Evaluate latency, correctness, scalability, resilience, and operational risk.

1. Event Streams and Streaming Architectures

An event is a record describing something that happened. Examples include OrderPlaced, TemperatureMeasured, AccountLogin, and PaymentAuthorized. A useful event normally contains:

• a unique event identifier;
• an event type;
• a payload containing business data;
• an event timestamp;
• a key identifying an entity, such as customer or account;
• a schema version and producer metadata.

An event stream is an ordered sequence of events, usually unbounded in duration. The word “ordered” requires qualification: a stream may be ordered within one partition or source, but not necessarily globally across a distributed system.

A streaming architecture commonly includes five logical layers:

1. Producers generate events from applications, devices, databases, or logs.
2. Brokers or event logs durably buffer and distribute events.
3. Stream processors transform, join, aggregate, filter, and enrich events.
4. State stores retain information needed across multiple events.
5. Sinks and consumers write results to databases, dashboards, alerts, or other topics.

This architecture separates event production from event consumption. Producers do not need to know which applications will use the data, and consumers can replay historical events when rebuilding a result.

Streaming and batch processing

Batch processing operates on a bounded collection. A streaming computation operates on an unbounded collection and therefore cannot normally wait for “all” input. It must define when a partial result is emitted and how that result may later be revised.

The distinction is conceptual rather than absolute. A streaming engine can process a bounded file as a finite stream, and a batch system can approximate near-real-time behavior by running frequently. The important questions are the required response time, data-arrival pattern, tolerance for incomplete information, and cost of continuous operation.

2. Brokers, Topics, and Partitions

A message broker receives records from producers and makes them available to consumers. Apache Kafka is often described as a distributed commit log. Records are appended to topics, and each topic is divided into partitions.

A partition is an append-only ordered log. Each record receives an offset, which identifies its position within that partition. Kafka does not provide one global ordering across all partitions. A producer may select a partition explicitly or use a record key. Records with the same key are commonly routed to the same partition, preserving per-key order as long as the partition assignment remains stable.

A consumer group is a set of consumers cooperating to read a topic. Within a group, a partition is normally assigned to one consumer at a time. Increasing partitions can increase parallelism, but it also introduces more coordination, storage, and ordering complexity.

Important broker design considerations include:

• Retention: how long records remain available for replay.
• Replication: how many broker copies exist.
• Replication factor: the number of replicas maintained for a partition.
• Consumer offset: the position a consumer has processed or committed.
• Backpressure: the mechanism by which a slow processor limits or signals upstream production.
• Schema compatibility: rules for evolving record structures without breaking consumers.

A broker is not merely a transport mechanism. Retention and replay make it possible to reconstruct state, test new consumers against historical data, and recover after processing failures.

3. Time Semantics

Time is one of the most difficult aspects of distributed stream processing because event creation, transmission, and processing occur at different moments.

Event time

Event time is when the event actually occurred according to the producer or source system. It is generally the most meaningful time for business analysis. For example, a vehicle location event should be grouped according to when the location was measured, not when it reached the processing cluster.

Ingestion time

Ingestion time is when a platform accepts an event. It is more consistent than producer clocks but may still differ significantly from event time when networks or upstream systems delay delivery.

Processing time

Processing time is when the stream processor handles the record. It is simple and often low-latency, but results can change when network delays or retries alter arrival patterns.

For a record, these times can be represented as:

This relationship is an idealized case, not a universal law. Clock errors, incorrect timestamps, clock correction, and backfilled records can violate it. The symbols represent timestamps for the same event, and all must use a consistent time scale and unit.

Event-time processing provides more reproducible business results, but requires mechanisms for handling out-of-order and late events.

4. Windows and Incremental Computation

Because a stream is unbounded, applications usually compute over windows. A window defines the subset of events included in a result.

Tumbling windows

A tumbling window divides time into non-overlapping intervals. A five-minute tumbling window contains events from 10:00:00 up to, but not including, 10:05:00, followed by the next interval. Tumbling windows are suitable for periodic counts, such as transactions per minute.

Sliding or hopping windows

A sliding window has a duration and an advance interval. For example, a 10-minute window evaluated every one minute produces overlapping results. Sliding windows provide smoother monitoring but require more computation because an event may contribute to multiple results.

Session windows

A session window groups activity separated by periods of inactivity. A web session might end after 30 minutes without an event. Session windows are useful when activity has no fixed duration, but sessions may remain open until inactivity or an explicit termination condition is observed.

Count-based windows

A count window contains a fixed number of records rather than a time interval. It can be useful when each record represents a comparable unit of work, although it is less directly related to business time.

A windowed aggregation can be expressed conceptually as:

Here, R(w, k) is the result for window w and key k, e is an event, key(e) returns its grouping key, and aggregate may be count, sum, average, minimum, maximum, or another associative operation. The expression assumes that event timestamps are valid and that the system has a policy for late data.

For an average, storing only the current average is insufficient for arbitrary incremental updates. A processor normally maintains count n and sum S, then computes:

When a new value x arrives, the state becomes S_new = S + x and n_new = n + 1. This method assumes the values are numeric and that numerical overflow and precision are managed appropriately.

5. State in Stream Processing

State is information retained across events. A stateless transformation, such as changing the format of a record, can process each event independently. A stateful transformation, such as counting orders per customer, depends on prior events.

Common state includes:

• a count or sum for each key;
• the latest value for each entity;
• a set of identifiers used for deduplication;
• partial results for a window;
• join buffers holding records from two streams;
• timers and session information.

State introduces several design problems. It must be partitioned consistently, persisted or recoverable, bounded where possible, and cleaned when no longer useful. A customer-keyed aggregation should route all events for a customer to the same logical state partition. If key distribution is highly uneven, one “hot key” can overload a task even when total traffic is moderate.

State growth and retention

Unbounded state is a common failure mode. A deduplication set that retains every identifier forever will eventually consume excessive storage. Designers should define a business retention period, use compact identifiers, expire inactive keys, or use probabilistic structures when approximate results are acceptable.

Stream joins

A stream-stream join usually requires state from both inputs. The processor retains records for a time range and matches records with compatible keys and timestamps. A join without bounds can require unbounded state and is therefore generally unsafe. A temporal or interval join should specify how long unmatched records remain eligible.

6. Watermarks and Late Data

A watermark is an operational assertion about event-time progress. In simplified form, a watermark W indicates that the processor expects few or no future events with timestamps less than or equal to W. It is not proof that no such event will ever arrive.

Suppose a five-minute event-time window ends at 12:05. The system may wait until a watermark reaches 12:05 before finalizing the window. If an event timestamped 12:03 arrives afterward, it is late relative to that window.

Late-data policies include:

1. Drop late data: simple and low-latency, but potentially inaccurate.
2. Update results: revise a previously emitted aggregate.
3. Route to a side output: preserve the event for audit or separate correction.
4. Use an allowed lateness period: keep state open for a specified duration.

The trade-off is fundamental: waiting longer improves completeness but increases latency and state retention. A watermark strategy should be based on observed delay distributions, source reliability, clock quality, and business tolerance for correction.

Watermarks are often generated per partition or input subtask and combined by the processor. In many systems, progress is constrained by the slowest active input partition. This protects correctness but means that a stalled partition can delay global event-time progress.

7. Delivery and Processing Semantics

Streaming systems must define what happens when failures occur between reading an input and committing an output.

At-most-once

At-most-once processing may lose records but avoids duplicates. A consumer can advance its input position before processing. This is appropriate only when occasional loss is acceptable, such as some telemetry or noncritical monitoring.

At-least-once

At-least-once processing retries or reprocesses records after uncertain failures. It avoids loss under the assumed failure model but may produce duplicates. Consumers should therefore be idempotent, meaning that repeating the same logical operation does not change the final result after the first successful application.

For example, setting an account status to SUSPENDED is usually more idempotent than incrementing a balance by 100 without a transaction identifier.

Exactly-once

Exactly-once is commonly used to mean that, for a defined source, computation, and sink boundary, each logical input affects the committed result once. It does not mean that an event exists only once everywhere in the system, nor does it automatically cover an arbitrary external API.

Exactly-once processing normally requires coordinated checkpoints, replayable input, deterministic or controlled computation, and transactional or idempotent output. If a processor writes to a database and acknowledges an input independently, a crash between those actions can still create duplicates unless the boundary is coordinated.

A practical design should document the guarantee precisely:

• Which records are covered?
• What happens during retries?
• Is the sink transactional?
• Can side effects such as email or payment calls be repeated?
• How are duplicates detected?

8. Fault Tolerance and Resilience

A resilient streaming system assumes that machines, networks, processes, and dependencies will fail. Important mechanisms include:

• replicated broker partitions;
• durable input retention;
• checkpoints of operator state and input positions;
• replay from a known consistent point;
• restart and recovery strategies;
• dead-letter or quarantine handling for malformed records;
• monitoring for lag, failures, and watermark progress.

A checkpoint is a consistent snapshot of relevant operator state and source positions. After failure, the processor restores the snapshot and replays subsequent input. Checkpoint intervals create a trade-off: frequent checkpoints reduce recovery work but consume more resources; infrequent checkpoints reduce overhead but increase recovery time and possible replay.

A simple estimate for recovery replay volume is:

If the input rate is 20,000 records per second and the last checkpoint was 30 seconds ago, the estimated replay is 600,000 records. This estimate assumes a stable rate and does not include backlog already present before the checkpoint.

Resilience also includes operational resilience. A system may recover technically but still fail its service objective if recovery takes too long, produces unacceptable corrections, or overwhelms downstream systems during catch-up.

9. Latency, Throughput, and Correctness

Streaming performance cannot be summarized by one number. Important measures include:

• End-to-end latency: time from event creation to usable output.
• Processing latency: time spent inside the processing topology.
• Throughput: records or bytes processed per unit time.
• Consumer lag: difference between the latest available input position and the consumer position.
• Watermark lag: difference between current wall-clock time and event-time progress.
• Backlog: amount of unprocessed input.
• Recovery time: time required to restore service after failure.
• Completeness: proportion of relevant events included in a result.

For an event, a basic latency decomposition is:

Each term is a nonnegative time component: source creation delay, transport delay, queueing delay, processing delay, and sink delay. The equation is an approximation and assumes the stages are measured along the same event path without double counting overlapping asynchronous work.

A low average latency does not guarantee good user experience. The 99th percentile may be much higher because of garbage collection, checkpointing, overloaded partitions, or slow downstream services. Engineers should therefore monitor distributions and service-level objectives rather than only averages.

Correctness also has dimensions:

• Completeness: relevant events are not omitted.
• Accuracy: transformations and aggregations are logically correct.
• Timeliness: results become available within the required interval.
• Consistency: related outputs do not contradict one another.
• Reproducibility: replay produces an equivalent result under the same assumptions.

Improving one dimension may harm another. Short watermarks reduce latency but increase late-data risk. Stronger delivery guarantees may increase coordination overhead. More retention improves replayability but raises storage cost.

10. Technology Patterns: Kafka, Flink, and Spark Structured Streaming

Kafka

Kafka is commonly used as the durable event backbone. Producers publish records to topics, and consumers read them by partition and offset. Kafka provides ordering within a partition, configurable retention, replication, and consumer-group coordination. It does not by itself perform complex event-time aggregation, maintain arbitrary application state, or guarantee exactly-once behavior across every external system.

Apache Flink

Flink is a distributed stream-processing engine designed for stateful computations over bounded and unbounded data. It provides event-time operations, windows, watermarks, managed state, timers, checkpoints, and recovery. Flink is especially suitable when precise event-time semantics, complex state, and continuous low-latency processing are central requirements.

Spark Structured Streaming

Spark Structured Streaming expresses streaming computations using a structured, DataFrame-like model. It supports incremental execution, event-time windows, watermarks, stateful operations, and integration with the broader Spark ecosystem. It is attractive where an organization already uses Spark for batch analytics, data lakes, and machine learning. Configuration and output-mode choices must be made carefully because not every operation supports every mode or guarantee.

A technology selection should consider:

These technologies are complementary rather than direct substitutes. A common architecture uses Kafka for ingestion and Flink or Spark for computation.

11. Designing a Streaming Solution

A disciplined design process begins with business and correctness requirements rather than with a product choice.

Step 1: Define the event contract

Specify event names, identifiers, keys, timestamps, units, schema versions, and compatibility rules. Avoid ambiguous timestamps such as time; use names such as occurred_at and document the time zone and precision.

Step 2: Define the service objective

State a target such as “95% of fraud alerts available within five seconds” and a correctness requirement such as “no accepted payment is charged twice.” Latency, availability, completeness, and cost should be explicit.

Step 3: Choose partitioning and parallelism

Select a key that supports required ordering and distributes load. Estimate peak traffic, record size, retention, and growth. Increasing partitions helps parallelism but does not solve a single hot key.

Step 4: Define time and window semantics

Choose event time when business meaning depends on occurrence time. Specify window type, duration, triggering behavior, watermark strategy, and allowed lateness.

Step 5: Design state and recovery

List every state variable, its key, expected cardinality, retention period, and recovery mechanism. Determine checkpoint frequency and acceptable replay time.

Step 6: Choose delivery semantics and sink behavior

Prefer idempotent operations where possible. If external side effects cannot be transactional, use an outbox, a deduplication key, or a workflow that separates computation from effect execution.

Step 7: Test adverse conditions

Test duplicates, out-of-order events, delayed partitions, malformed records, schema changes, broker failure, processor restart, sink unavailability, hot keys, and traffic spikes. A streaming system is not production-ready merely because it works on an ideal continuous input.

12. Illustrative Case: Real-Time Payment Fraud Detection

Background

A fictional online retailer, Northstar Market, processes card payments globally. Its existing fraud report runs every hour, allowing suspicious purchases to remain active for too long. The retailer wants an alert within three seconds for high-risk activity.

Problem

The system receives payment attempts, account logins, device observations, and chargeback events. Events may arrive out of order because mobile devices and regional services have different network conditions. A customer may legitimately use several devices, while a stolen account may generate rapid purchases from distant locations.

Analysis

The team identifies three requirements:

1. Recent payment velocity must be calculated per account.
2. Device and geographic anomalies must be combined with payment history.
3. Duplicate alerts must not cause repeated account-lock actions.

A five-minute sliding event-time window is selected for payment velocity. A short allowed-lateness interval is used because most events arrive within two seconds, but a small number arrive later. A separate state store retains recent device observations for each account.

Approach and solution

Payment and login events are published to Kafka topics. The account identifier is used as the partition key for account-specific ordering. A Flink job consumes both topics, validates schemas, filters impossible timestamps, and maintains keyed state.

For each account, the job computes payment count and total value in the event-time window. It joins recent login and device information using a bounded time interval. The result is assigned a risk score according to a versioned business rule. If the score exceeds a threshold, the job emits an alert containing the account identifier, triggering event identifiers, rule version, and a unique alert key.

The alert sink uses an idempotency key derived from the account and triggering condition. Account-lock requests are not sent directly from an uncoordinated operator; they pass through a service that records processed alert keys transactionally with the lock decision. This limits repeated effects during retries.

Watermarks allow the processor to finalize most event-time windows while a late-event stream preserves records that arrive beyond the allowed interval. Operators monitor Kafka lag, processing latency, watermark delay, checkpoint duration, alert rate, and duplicate suppression.

Outcome

In this illustrative design, the retailer can produce rapid alerts while retaining a correction path for late events. The design does not claim perfect fraud detection. It explicitly manages false positives, clock errors, incomplete data, and repeated side effects. Analysts can replay retained Kafka topics with a new rule version to evaluate changes before deployment.

Learning points

• A low-latency alert is not sufficient if duplicate actions can harm customers.
• Event-time windows provide more meaningful fraud metrics than processing-time windows when network delay is variable.
• State must be bounded and recoverable.
• The business rule, schema version, and input event identifiers should accompany decisions for auditability.
• Human review remains important for high-impact account actions.

13. Governance, Security, and Ethical Issues

Streaming systems often process personal, financial, health, or location data. Governance should therefore be designed into the pipeline.

Data minimization limits collection and retention to what the purpose requires. Access control should apply to topics, consumer groups, state stores, logs, and replay tools. Encryption should protect data in transit and at rest. Sensitive values should be tokenized or masked in operational logs.

Replay creates a governance concern: a retained event may be reused long after its original purpose. Retention periods, deletion procedures, legal holds, and subject-access requirements must be considered. Deleting a record from a broker may not remove copies from state stores, backups, derived tables, or downstream caches.

Automated streaming decisions can produce unfair outcomes. Fraud or risk models may disproportionately flag certain groups because of biased historical data or proxy variables. Systems should support explanation, review, appeal, model monitoring, and versioned decision records. In regulated contexts, auditability includes knowing which input events, rules, model version, and threshold produced an outcome.

14. Challenges and Limitations

Streaming systems are powerful but not universally appropriate. Continuous operation increases infrastructure and monitoring costs. Debugging is harder because failures may depend on timing, partition assignment, or event order. Exactly-once claims can be misunderstood when external side effects are involved. Watermarks cannot repair fundamentally incorrect source timestamps. Stateful jobs may accumulate large recovery snapshots, and schema changes can break old consumers.

Approximate algorithms may reduce memory use for very high-cardinality streams, but they introduce error that must be measured and communicated. Machine-learning models in streams may suffer from concept drift, delayed labels, and feedback loops. Finally, not every decision needs real-time processing: if a daily report meets the business need, a simpler batch architecture may be more reliable and economical.

15. Best Practices

1. Give every event a stable identifier and an explicit event timestamp.
2. Treat schemas as contracts and manage compatibility deliberately.
3. Partition according to required ordering and expected load distribution.
4. Prefer event time for business windows when source timestamps are trustworthy.
5. Define watermark and late-data behavior before implementation.
6. Bound state with retention, timeouts, or compaction.
7. Make sinks idempotent or use transactional coordination where possible.
8. Monitor percentile latency, lag, watermark progress, checkpoint health, and error rates.
9. Test replay, duplicates, disorder, failure, backpressure, and schema evolution.
10. Document exactly what “exactly once” means at each system boundary.
11. Protect sensitive data during storage, processing, logging, and replay.
12. Keep a correction and audit path for high-impact decisions.

Chapter Summary

Real-time data systems represent business activity as event streams and process those streams incrementally. Brokers such as Kafka provide durable, partitioned, replayable logs, while engines such as Flink and Spark Structured Streaming perform stateful transformations, windows, joins, and aggregations. Partitions provide scalability and local ordering but do not create a universal global order.

Event time, ingestion time, and processing time have different meanings. Event-time windows require watermarks and explicit policies for late data. Stateful processing supports useful operations such as keyed aggregation, deduplication, and stream joins, but state must be bounded, partitioned, and recoverable. At-most-once, at-least-once, and exactly-once semantics describe different failure behaviors; exactly-once is meaningful only within a defined system boundary.

A professional streaming design evaluates latency, throughput, completeness, correctness, recovery, cost, security, and governance together. Reliable systems use durable inputs, checkpoints, idempotent effects, observability, replay testing, and clear event contracts. Real-time processing is valuable when timely action matters, but it should be chosen because its benefits justify its complexity.

Key Terms

Event: A record describing something that happened.

Event stream: An unbounded, continuously produced sequence of events.

Broker: A service that receives, stores, and distributes records.

Topic: A named logical stream of records in a broker.

Partition: An ordered append-only subdivision of a topic.

Offset: A record position within a partition.

Consumer group: Consumers cooperating to process a topic’s partitions.

Event time: The time at which an event occurred at its source.

Processing time: The time at which a processor handles an event.

Watermark: An estimate of progress in event time.

Window: A rule defining which events contribute to a result.

Tumbling window: A non-overlapping, fixed-duration window.

Sliding window: An overlapping window evaluated at regular advances.

Session window: A window formed by periods of activity separated by inactivity.

State: Information retained across events.

Checkpoint: A recoverable snapshot of state and input positions.

At-least-once: Processing that avoids loss under assumptions but may duplicate effects.

Exactly-once: A defined guarantee that each logical input affects a committed result once within a specified boundary.

Backpressure: A mechanism that limits upstream production when downstream processing is overloaded.

Idempotence: The property that repeating an operation does not change its intended final effect.

Review Questions

5 Short Answer Questions

1. What is the difference between event time and processing time?
2. Why does Kafka partition a topic?
3. What purpose does a watermark serve?
4. Why must state be bounded in a long-running streaming application?
5. Define at-least-once processing.

5 Long Answer Questions

1. Explain the architecture of a Kafka-based streaming system from producer to sink.
2. Compare tumbling, sliding, session, and count-based windows, including suitable applications.
3. Explain how checkpoints support fault tolerance and discuss their performance trade-offs.
4. Compare at-most-once, at-least-once, and exactly-once semantics with examples of suitable use cases.
5. Discuss the trade-off among latency, completeness, state retention, and late-data handling.

3 Analytical / Application Questions

1. Design an event-time streaming solution for detecting unusual login activity. Specify event fields, partition key, window, watermark policy, state, delivery semantics, and monitoring metrics.
2. A stream receives 12,000 records per second, and the last successful checkpoint occurred 45 seconds before a failure. Estimate the replay volume and explain factors that could make the actual value different.
3. A business wants a five-minute dashboard updated every 30 seconds, but 8% of events arrive more than two minutes late. Evaluate whether a two-minute watermark delay is appropriate and propose an alternative late-data policy.

Multiple Choice Questions with Answers

Question 1

Which ordering guarantee does Kafka generally provide?

A. A single global order across every topic

B. Ordering of records within a partition

C. Ordering of records across all consumer groups

D. Ordering based automatically on event time

Correct Answer: B. Ordering of records within a partition

Question 2

Which time concept represents when an event actually occurred at its source?

A. Processing time

B. Ingestion time

C. Event time

D. Checkpoint time

Correct Answer: C. Event time

Question 3

What is the main purpose of a watermark?

A. To encrypt records in transit

B. To estimate progress in event time

C. To increase the number of partitions

D. To delete all late records immediately

Correct Answer: B. To estimate progress in event time

Question 4

Which window type does not overlap adjacent intervals?

A. Sliding window

B. Session window

C. Tumbling window

D. Interval join window

Correct Answer: C. Tumbling window

Question 5

Which processing guarantee may produce duplicate effects after a retry?

A. At-most-once

B. At-least-once

C. Exactly-once

D. No-delivery processing

Correct Answer: B. At-least-once

Question 6

A stream processes 12,000 records per second. Approximately how many records may need replay after 45 seconds without a successful checkpoint?

A. 54,000

B. 120,000

C. 540,000

D. 5,400,000

Correct Answer: C. 540,000

Question 7

Which design most directly limits duplicate external actions during retries?

A. Using an idempotency key at the sink

B. Removing all checkpoints

C. Increasing window duration indefinitely

D. Using processing time instead of event time

Correct Answer: A. Using an idempotency key at the sink

Question 8

What is a major risk of an unbounded deduplication state store?

A. It prevents all event ordering

B. It eventually consumes excessive storage

C. It guarantees data loss

D. It eliminates the need for schemas

Correct Answer: B. It eventually consumes excessive storage

Question 9

Which technology is primarily a durable event log and transport layer?

A. Kafka

B. Flink

C. Spark Structured Streaming

D. A watermark generator

Correct Answer: A. Kafka

Question 10

If a late event arrives after a window has been finalized, which policy preserves it for separate review or correction?

A. Discarding it without recording

B. Routing it to a side output

C. Reducing the partition count

D. Disabling event timestamps

Correct Answer: B. Routing it to a side output