Stream and Real-Time Processing

Introduction

Modern information systems increasingly operate on continuously arriving data rather than periodically collected files. Payment authorisation, industrial sensors, connected vehicles, cybersecurity telemetry, online recommendations and social-media monitoring all require decisions while events are still fresh. Stream processing is the discipline of ingesting, transforming, correlating and analysing unbounded sequences of events with low and predictable latency.

A batch job asks, “What happened during the last processing interval?” A streaming application asks, “What is happening now, and what should the system do next?” The distinction is not merely performance-related. Streaming systems must reason about event time, incomplete information, disorder, failures, duplicates and changing state.

This chapter develops the engineering principles needed to build event-driven pipelines with Apache Kafka and stream-processing frameworks such as Apache Flink, Spark Structured Streaming and Kafka Streams. Particular attention is given to windows, watermarks, state, late data and delivery semantics.

Learning Objectives

After studying this chapter, a learner should be able to:

1. Explain the architecture and operational role of Kafka in event-driven systems.
2. Distinguish processing time, ingestion time and event time.
3. Select suitable windows and watermarks for analytical requirements.
4. Design stateful operators and reason about checkpointing and recovery.
5. Compare at-most-once, at-least-once and exactly-once processing.
6. Handle late, duplicate and out-of-order events explicitly.
7. Evaluate the performance, correctness, governance and operational risks of a real-time pipeline.

1. The Stream-Processing Model

A stream is an ordered sequence of records, potentially unbounded in duration. A record normally contains a key, payload, event timestamp, source metadata and sometimes a schema version. A useful abstraction is:

where k_i is the record key, v_i its value, t_i^e its event-time timestamp and m_i metadata. The sequence may be ordered within a partition but not globally ordered across all producers or partitions.

A stream-processing topology commonly contains:

• Sources, which read events from brokers, databases, files or APIs.
• Transformations, such as filtering, mapping, enrichment and joins.
• Stateful operators, such as aggregations, deduplication and pattern detection.
• Sinks, which write results to databases, topics, object stores or applications.
• Coordination and recovery services, including checkpoints, offset management and monitoring.

The central design tension is between latency, completeness and resource usage. A result produced immediately may omit events that arrive later. Waiting longer improves completeness but increases latency and retained state.

2. Kafka as an Event Backbone

Kafka is a distributed commit-log platform. Producers append records to named topics. Topics are divided into partitions, and each record receives an offset within its partition. Consumers read records and commit their progress. A consumer group distributes partitions among its members, allowing horizontal scaling.

Partitioning has three important consequences:

1. Records with the same key can be routed to the same partition, preserving per-key order.
2. Parallelism is bounded by the number of partitions assigned to a processing stage.
3. Kafka does not, by itself, provide a total order across partitions.

A partition is therefore an ordering boundary, not merely a storage unit. Choosing a key such as account ID, device ID or order ID is both a scalability decision and a correctness decision. A poor key can create a hot partition, while a changing or absent key can make per-entity state difficult to maintain.

Kafka provides durable retention, replay and consumer-group coordination. Retention is distinct from business correctness: retaining an event does not guarantee that a downstream database will not receive a duplicate. Producers may use idempotence, transactions and acknowledgements to reduce duplication, but the complete delivery guarantee depends on the entire pipeline.

A typical architecture is:

Figure: A Kafka-backed event-driven processing architecture.

3. Time Semantics

Time is one of the most important design choices in stream processing.

• Event time is when an event actually occurred at its source. It is normally carried in the record.
• Ingestion time is when a platform accepts the event.
• Processing time is when an operator evaluates the event.

Processing time is simple and often fast, but results can vary with network delay and workload. Event time gives more reproducible business results, particularly when devices buffer data or networks are unreliable. It requires timestamps, a policy for disorder and a mechanism to decide when sufficient data has arrived.

For example, a vehicle may produce an event at 10:00:05, upload it at 10:00:20 and have it processed at 10:00:21. These are three different times. A five-minute event-time average should classify the event using 10:00:05, not the later arrival times.

4. Windows and Continuous Aggregation

Because a stream is unbounded, an aggregation over the entire stream cannot normally be held in finite memory. A window limits the portion of the stream considered at one time.

4.1 Tumbling windows

A tumbling window has a fixed size and no overlap. A 60-second window may cover [10:00:00, 10:01:00), followed by [10:01:00, 10:02:00). Half-open intervals avoid assigning a boundary event to two windows. For a sum, the window result is:

Here w is a window, t_i^e is event time and x_i is the measured value. This formula assumes the aggregation includes each qualifying event once and that the value is additive.

Tumbling windows suit billing intervals, periodic operational reports and non-overlapping counts.

4.2 Sliding or hopping windows

A sliding window has a length larger than its advance or slide. A 10-minute window evaluated every minute overlaps previous windows. It gives smoother monitoring but may process an event multiple times across outputs and requires more state.

4.3 Session windows

A session window groups activity separated by no more than a configured inactivity gap. It is useful for user sessions and bursts of device activity. Session windows are data-dependent: a later event can merge two previously separate sessions, which makes late-data handling especially important.

4.4 Global and keyed windows

A global window covers all records, whereas a keyed window maintains an independent window for each key. Keyed windows support per-customer, per-device or per-account analysis. They can nevertheless create severe state growth if the key cardinality is large or keys are never retired.

5. Watermarks and Late Data

A watermark is an estimate that the system has probably observed events up to a particular event-time boundary. It is not proof that no earlier event will ever arrive. A common conceptual form is:

where W_p is the watermark for partition p and δ is an allowed-out-of-orderness duration. The expression assumes timestamps are comparable, the partition’s observed maximum is meaningful and the delay policy is fixed or otherwise explicitly managed.

A window can be considered complete when its end is earlier than the relevant watermark. In a multi-partition operator, progress is generally constrained by the least-advanced input partition; otherwise a fast partition could close a join or aggregate before a slow partition contributes matching data.

The parameter δ is a business and engineering choice:

• A small value reduces latency and memory but increases late results.
• A large value improves completeness but delays output and retains state.
• A fixed value may be inappropriate when delay distributions change.

Late events should not be silently discarded. Common policies include:

1. Drop events beyond the allowed lateness, with metrics and a dead-letter stream.
2. Update a previously emitted result and propagate a correction downstream.
3. Route late events to a side output for reconciliation.
4. Reopen state for a bounded period, if the sink and consumers support revisions.

The correct policy depends on whether results are provisional, append-only or legally/auditably final.

A useful operational measure is event lateness:

where L_i is lateness, t_i^p is processing or arrival time and t_i^e is event time. This assumes both timestamps use the same clock basis and units. In practice, clock synchronisation and timestamp quality must be monitored.

6. Stateful Processing

Stateless transformations process each record independently. Stateful operators retain information across records, including counts, running totals, join buffers, timers, deduplication identifiers and machine-learning features.

State should normally be partitioned by key. For a keyed count:

where C_k,t is the count for key k before the event and t identifies the update step. The update assumes each input event is counted once. Under at-least-once delivery, that assumption may fail unless duplicates are removed or the update is idempotent.

State-management requirements include:

• Durability: state must survive process failure.
• Consistency: state and source progress must recover to a compatible point.
• Boundedness: windows, time-to-live policies and cleanup prevent unbounded growth.
• Serialization: state schemas must evolve safely.
• Partition mobility: rebalancing must move state efficiently.
• Observability: state size, checkpoint duration and restoration time require metrics.

Flink uses checkpoints and state backends; Kafka Streams commonly uses changelog topics and local state stores; Spark Structured Streaming uses checkpoint locations to record progress and state. Their APIs differ, but the underlying principle is similar: durable progress and durable state must be coordinated.

7. Delivery and Processing Semantics

Delivery semantics describe how records and effects behave when failures occur.

At-most-once usually commits progress before processing. A crash after the commit but before the output loses the record. At-least-once processes before committing progress; a crash between output and commit can cause replay and duplication.

Exactly-once is not a magical property of a whole business process. It requires coordinated mechanisms such as transactional source-to-sink writes, deterministic replay, idempotent effects or an outbox/inbox design. Kafka transactions can atomically commit produced records and consumed offsets in a Kafka-to-Kafka topology. An external database requires a compatible transaction, an idempotency key, or a carefully designed connector.

An idempotent sink makes repeated application of the same logical operation equivalent to one application. For example, replacing a record by a unique transaction ID is safer than blindly incrementing a balance. However, idempotence must be designed at the business-operation level; merely retrying a non-idempotent API is not exactly-once.

8. Joins, Enrichment and Event-Time Correlation

Streaming joins combine records from multiple streams or a stream and a bounded reference table. A join must define:

• the matching key;
• the temporal relationship;
• how long unmatched records are retained;
• what happens when a match arrives late;
• how updates and deletions are represented.

A time-bounded join might match events when their timestamps differ by no more than Δt. Retaining unlimited unmatched records is unsafe, so watermarks and state time-to-live policies are essential. Reference data can be cached, but cache staleness must be measured and governed.

9. Performance and Capacity Reasoning

A real-time design should quantify throughput and latency rather than rely on labels such as “low latency.” If an input rate is λ records per second and average record size is s bytes, raw ingress is:

where R is bytes per second. Replication, headers, compression and serialization add overhead, so storage and network capacity must exceed the raw value.

For a window retaining events for duration T at rate λ, a rough event-retention estimate is:

This assumes a stable rate and one logical stream. For keyed state, multiply by the number of active keys and include aggregation metadata. The estimate is not a sizing guarantee: bursts, skew, replication and checkpoint copies require safety margins.

Important service-level indicators include end-to-end event latency, consumer lag, watermark lag, late-event percentage, throughput, checkpoint duration, restart time, state size, error rate and duplicate rate. Averages alone conceal tail behaviour; percentile latency, such as p95 or p99, is often more useful.

10. Implementation Pattern

A robust implementation typically follows this sequence:

1. Define the event contract: identifiers, event timestamp, schema, producer and version.
2. Select a partition key that supports ordering and distributes load.
3. Validate records at ingress and isolate malformed records.
4. Assign event-time timestamps and a documented out-of-orderness policy.
5. Apply keyed windows or timers with explicit state cleanup.
6. Choose a delivery guarantee based on business impact, not marketing terminology.
7. Make sinks idempotent or transactional where retries are possible.
8. Instrument lag, watermarks, state, late data and corrections.
9. Test crashes, replays, partition skew, clock errors and schema evolution.
10. Define operational runbooks for backfills, replay and disaster recovery.

Framework selection should reflect workload characteristics. Kafka Streams is attractive for Kafka-centric applications and embedded library deployment. Flink is strong for sophisticated event-time processing, large state and complex stream topologies. Spark Structured Streaming integrates naturally with lakehouse and batch analytics, although latency and state behaviour depend on the selected execution mode and workload. No framework eliminates the need for sound data contracts and failure design.

11. Illustrative Case: Real-Time Fraud Detection

Background

A fictional digital bank receives card-authorization events from mobile and point-of-sale systems. It wants to flag a card when three or more purchases occur in ten event-time minutes and the total exceeds £500, while tolerating mobile-network delays of up to two minutes.

Problem

Events can arrive out of order, the same authorization may be retried, and a fraud decision must be available quickly. A false negative is costly, but a false positive creates customer friction. The system must also preserve an auditable explanation.

Analysis

The authorization ID is a deduplication key, while card ID is the aggregation key. A tumbling window could miss activity spanning a boundary; a sliding window of ten minutes evaluated frequently is more appropriate. A two-minute watermark allowance balances completeness and decision latency. The rule is provisional until the watermark passes the relevant event-time boundary, but urgent alerts may be emitted earlier and later corrected.

Approach

Producers publish validated events to Kafka, keyed by card ID. The processor removes repeated authorization IDs, assigns event time from the issuer timestamp and maintains per-card state. It computes count and total within the sliding window, emits an alert when both thresholds are exceeded, and writes the decision plus contributing authorization IDs to an audit topic. State expires after the maximum window plus allowed lateness. A transactional Kafka sink ensures that the result and consumed offset are committed together; downstream case-management writes use the alert ID as an idempotency key.

Outcome

The design supports replay, explains each alert and limits state. Late events can produce a correction event rather than silently changing history. Operational dashboards expose watermark lag, late-event rates, duplicate authorizations and alert latency.

Learning points

The case demonstrates that fraud detection is not merely a windowed count. Correctness depends on key selection, event-time quality, deduplication, lateness policy, state expiry, delivery semantics and auditability.

12. Governance, Security and Ethical Considerations

Streaming pipelines process personal, financial, location and behavioural data. Governance should include data minimisation, purpose limitation, retention schedules, access control, encryption, schema ownership and lineage. Sensitive fields should be tokenised or masked where feasible. Access to raw topics and state stores should be audited.

Automated decisions require special care. A low-latency fraud or credit rule may disproportionately affect particular groups if input data or thresholds encode bias. Organisations should document features, decision rules, override procedures, appeal mechanisms and model monitoring. Real-time processing does not remove obligations under privacy, financial-services, cybersecurity or records-management regulations.

AI models can enrich streams with anomaly scores or predictions, but model version, feature timestamp, training-data lineage and confidence should accompany each decision. A model score must not be treated as an unquestionable fact. Drift, feedback loops and delayed labels require continuous evaluation.

13. Challenges and Contemporary Developments

Major engineering challenges include skewed keys, burst traffic, poison messages, schema incompatibility, clock drift, long recovery times, state explosion and the semantic gap between a streaming result and an external side effect. Exactly-once claims are often overstated when a pipeline crosses systems that do not share a transaction boundary.

Contemporary architectures increasingly combine Kafka-like logs with lakehouse storage, allowing streaming and batch consumers to share governed data. Event-driven microservices use topics to decouple producers and consumers, but excessive topic coupling can make contracts difficult to evolve. Adaptive watermarks, tiered storage, incremental materialized views and stream-native feature stores are active areas of practice and research.

The strongest designs remain explicit about boundaries: what is guaranteed, at what time, for which key, against which failures and with what correction mechanism.

Best-Practice Checklist

• Use event IDs and immutable event timestamps.
• Preserve per-entity ordering through stable partition keys.
• Treat watermarks as estimates, not proofs.
• Measure and govern late data rather than hiding it.
• Bound every state store with windows, timers or retention.
• Prefer idempotent effects and transactional boundaries.
• Separate provisional, corrected and final results in the data contract.
• Test replay, failure, duplication, skew and delayed events.
• Monitor tail latency and watermark progress.
• Document privacy, retention, lineage and model accountability.

Chapter Summary

Stream processing handles unbounded event sequences and enables decisions with low latency. Kafka supplies durable partitioned logs, replay and consumer-group coordination, while frameworks such as Flink, Kafka Streams and Spark Structured Streaming provide transformations, windows and state. Event time is generally preferable for business correctness, but it requires watermarks and explicit late-data policies. Tumbling, sliding and session windows provide different analytical semantics and resource costs. Stateful processing must be bounded, durable and recoverable. At-least-once processing is common and requires deduplication or idempotent sinks; exactly-once is meaningful only within clearly defined transactional boundaries. Successful systems combine technical design with capacity planning, observability, security, privacy and explainable operational policy.

Key Terms

Event stream: An unbounded sequence of records produced over time.

Kafka topic: A named durable log to which records are published.

Partition: An ordered Kafka subdivision that enables parallelism.

Offset: A record position within a Kafka partition.

Consumer group: Consumers cooperating to process topic partitions.

Event time: The time at which an event occurred at its source.

Processing time: The time at which a processor handles an event.

Watermark: An estimate of event-time progress.

Lateness: The difference between processing/arrival time and event time.

Tumbling window: A fixed, non-overlapping window.

Sliding window: An overlapping window evaluated at a configured advance.

Session window: A window separated by an inactivity gap.

Stateful operator: An operator that retains information across records.

Checkpoint: A consistent recovery point for progress and state.

At-most-once: Processing that may lose records but normally avoids retries.

At-least-once: Processing that avoids intentional loss but may duplicate effects.

Exactly-once: A guarantee that each logical result affects a defined transactional boundary once.

Idempotence: Repeating an operation produces the same logical outcome as one application.

Consumer lag: The difference between available source progress and consumer progress.

Schema evolution: Controlled change of event structure across producer and consumer versions.

Review Questions

Short Answer Questions

1. Why is Kafka partitioning both a scalability and a correctness decision?
2. Distinguish event time from processing time.
3. What purpose does a watermark serve?
4. State one advantage and one cost of a sliding window.
5. Why does at-least-once processing require idempotence or deduplication?

Long Answer Questions

1. Explain Kafka’s topic, partition, offset and consumer-group model and relate each to stream-processing architecture.
2. Compare tumbling, sliding and session windows, including suitable applications and state implications.
3. Discuss how watermarks, allowed lateness and late-event policies affect completeness and latency.
4. Compare at-most-once, at-least-once and exactly-once semantics, including the limitations of exactly-once across external systems.
5. Develop a state-management and observability plan for a high-volume event-time pipeline.

Analytical / Application Questions

1. A monitoring system receives 2,000 events per second and retains a ten-minute window. Estimate the number of retained events under a stable-rate assumption, and identify two factors that require additional capacity.
2. Design a stream pipeline for detecting three failed logins for an account within five minutes when events may be delayed by 30 seconds. Specify the key, window, watermark and late-data policy.
3. Critically evaluate whether a payment-alert system should drop late events, revise previous results or route them for reconciliation. Justify the choice using business risk, auditability and latency.

Multiple Choice Questions with Answers

Question 1

Which Kafka feature primarily enables parallel processing of a topic?

A. Partitions

B. Schema registry

C. Message headers

D. Consumer offsets

Correct Answer: A. Partitions

Question 2

A record occurs at 10:00:05, is ingested at 10:00:20 and processed at 10:00:21. Which is its event time?

A. 10:00:05

B. 10:00:20

C. 10:00:21

D. The time of window closure

Correct Answer: A. 10:00:05

Question 3

What is the principal purpose of a watermark?

A. To encrypt records

B. To estimate event-time progress

C. To assign Kafka partitions

D. To compress state

Correct Answer: B. To estimate event-time progress

Question 4

Which window is non-overlapping by definition?

A. Session

B. Sliding

C. Tumbling

D. Global keyed

Correct Answer: C. Tumbling

Question 5

Under at-least-once processing, which outcome is possible?

A. Records can be duplicated after recovery

B. Records are always lost

C. No retry can occur

D. External effects are automatically transactional

Correct Answer: A. Records can be duplicated after recovery

Question 6

For a stable input rate of 2,000 events per second and a ten-minute window, the approximate retained event count is:

A. 12,000

B. 120,000

C. 1,200,000

D. 2,000,000

Correct Answer: C. 1,200,000

Question 7

Which key would normally support per-account ordering and state?

A. Random number per event

B. Account ID

C. Processor hostname

D. Window end time

Correct Answer: B. Account ID

Question 8

What is a suitable response to a late event when auditability requires correction?

A. Silently discard it

B. Overwrite history without notification

C. Emit a correction or reconciliation result

D. Change the event timestamp at random

Correct Answer: C. Emit a correction or reconciliation result

Question 9

Why must state be bounded in a long-running stream application?

A. Unbounded state can exhaust storage and memory

B. Bounded state prevents all replay

C. State is never needed for joins

D. Watermarks eliminate all state

Correct Answer: A. Unbounded state can exhaust storage and memory

Question 10

Which design most directly makes repeated sink operations safe?

A. Using a unique idempotency key

B. Increasing window overlap

C. Removing event timestamps

D. Reducing Kafka partitions to one

Correct Answer: A. Using a unique idempotency key