Distributed Data Processing and Analytics

Introduction and Context

Modern organisations generate data continuously from transactional systems, mobile devices, sensors, web applications, scientific instruments and business processes. The value of this data depends not only on storage, but also on the ability to transform, correlate and analyse it at scale. Distributed data processing divides computation across multiple networked machines so that very large datasets, high event rates and complex analytical workloads can be handled within acceptable time and cost limits.

Distributed analytics is not simply “running a program on many computers.” It requires decisions about data partitioning, communication, scheduling, fault recovery, consistency, event time, state, data quality and operational governance. A system may process a petabyte-scale historical dataset efficiently but fail to provide timely results for live events. Conversely, a low-latency stream processor may produce rapid but unreliable results if late, duplicated or incorrectly ordered events are not handled.

This chapter examines four important processing models: MapReduce, dataflow, actors and event streams. It then connects these models to batch, streaming and graph-processing pipelines. Particular attention is given to locality, partitioning, windowing, state management, exactly-once claims, data quality and real-time analytics.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the architecture and execution principles of MapReduce, dataflow, actor and event-stream models.

2. Select suitable processing models for batch, streaming and graph workloads.

3. Evaluate partitioning strategies, data locality, skew and communication cost.

4. Design windowed streaming computations and reason about state, time and late data.

5. Critically evaluate claims of exactly-once processing.

6. Design data-quality controls and governance mechanisms for distributed analytics.

7. Compare graph-processing approaches and identify their operational limitations.

8. Assess real-time analytics architectures using correctness, latency, throughput and cost.

1. Distributed Processing as a Systems Problem

A distributed processing pipeline generally contains five logical stages:

• Ingestion: acquiring data from files, databases, sensors, applications or message brokers.
• Partitioning: assigning records to workers or storage partitions.
• Transformation: filtering, mapping, joining, aggregating or enriching records.
• Materialisation: writing results to files, tables, indexes, dashboards or downstream services.
• Coordination and recovery: scheduling tasks, tracking progress, handling failures and managing metadata.

The central trade-off is between computation, communication and correctness. Computation can often be parallelised, but data-dependent operations such as joins and global aggregations require records to meet at the same worker. Network transfer is slower and less predictable than local memory or local storage access. Therefore, a good distributed design attempts to place computation near its data, reduce unnecessary movement and make intermediate results recoverable.

Important performance measures include:

• Throughput: records, bytes or jobs processed per unit time.
• Latency: time from data arrival or job submission to result availability.
• Scalability: ability to maintain useful performance as data or resources increase.
• Availability: ability to continue serving or processing despite failures.
• Correctness: whether results satisfy the intended semantics.
• Cost efficiency: useful processing achieved per unit of infrastructure or energy.

These measures can conflict. Replicating state may improve availability but increase cost. Stronger consistency may increase latency. Aggressive parallelism may reduce execution time until coordination and network overhead dominate.

2. Processing Models

2.1 MapReduce

MapReduce is a programming and execution model designed primarily for large-scale batch processing. An input dataset is divided into splits. Mapper tasks independently transform input records into intermediate key-value pairs. A shuffle phase groups records with the same key and transfers them to reducers. Reducers then aggregate or transform each key’s values and write output.

Conceptually:

```text

map(record) -> [(key, value)]

shuffle: group all values by key

reduce(key, [values]) -> result

```

For example, word counting maps each document into pairs such as `(distributed, 1)`. The shuffle groups all values associated with `distributed`, and the reducer sums them.

MapReduce is attractive because it provides automatic parallelism, task scheduling and recovery. If a worker fails, its task can be rerun. Intermediate data is usually materialised, creating a fault boundary between stages. This design is robust for long-running batch jobs over large datasets.

Its limitations include high latency caused by job startup and materialisation, difficulty expressing iterative algorithms and potentially expensive shuffles. Algorithms involving repeated graph traversals or machine-learning iterations may perform poorly if every iteration writes and rereads data.

A combiner may perform local pre-aggregation before shuffle. For word counting, a mapper can locally combine several `(word, 1)` pairs into `(word, count)` pairs. A combiner is an optimisation, not a replacement for the reducer: it must be safe to execute zero, one or several times. Thus, combiners are most appropriate for associative and commutative aggregation operations.

2.2 Dataflow Systems

A dataflow model represents computation as a directed graph of operators connected by data streams. Operators may include filters, projections, joins, aggregations, windows and sinks. Systems such as modern batch-stream engines often express both bounded data and unbounded data using the same logical abstractions.

A dataflow graph may be:

• Acyclic, as in a simple extract-transform-load pipeline.
• Iterative, as in machine learning or graph algorithms.
• Streaming, where records continuously traverse operators.
• Hybrid, where historical data is processed in batch and new events are processed continuously.

Dataflow systems can optimise the graph by pipelining operators, selecting join strategies, fusing compatible operations and inserting checkpoints. They often distinguish between logical time, which describes the semantics of the computation, and physical processing time, which describes when a machine executes it.

Dataflow is more expressive than classic MapReduce, but this flexibility creates more design choices. A developer must understand operator parallelism, state growth, checkpoint frequency, backpressure and the semantics of retries.

2.3 Actor Model

In the actor model, an actor is an independent computational entity with private state and a mailbox. Actors communicate by sending asynchronous messages rather than directly sharing mutable memory. Upon receiving a message, an actor may modify its state, send messages, create actors or change its future behaviour.

The actor model is useful for applications containing many independent entities, such as user sessions, devices, accounts, game objects or workflow instances. It supports location transparency: an actor may be local or remote, although network failures mean that remote communication cannot be treated as equivalent to local method invocation.

Actor systems provide a natural model for concurrency and isolation. However, global analytics are not automatically easy. Computing a total across millions of actors requires aggregation, routing or a separate analytical system. Mailbox ordering, supervision, retries and duplicate messages must also be specified.

2.4 Event-Stream Model

An event-stream architecture treats events as durable facts published to an append-oriented log. Producers write events, and consumers independently read them, often retaining offsets that identify progress. Multiple consumers can process the same events for different purposes, such as fraud detection, reporting and machine-learning feature generation.

This model separates event production from consumption and supports replay. Replay is valuable for correcting a bug, rebuilding a materialised view or adding a new consumer. It also makes event schemas, retention policies and ordering rules important architectural concerns.

An event is not necessarily the same as the current state of an entity. For example, `AccountCredited` and `AccountDebited` are events; the current balance is derived state. Event sourcing deliberately stores state-changing events as the primary record, whereas conventional systems may store only the latest state and emit events as secondary outputs.

3. Batch, Stream and Graph Processing

3.1 Batch Processing

Batch processing operates on a finite, identifiable dataset. It is appropriate for historical reporting, data-lake transformations, regulatory extracts, model training and periodic aggregation. Batch jobs can exploit large sequential reads, extensive planning and cheap recomputation.

A typical batch pipeline reads immutable source data, validates schema and quality, performs transformations and joins, and writes partitioned analytical tables. Good practice includes deterministic transformations, immutable raw data, versioned code and metadata describing input and output datasets.

Batch processing is not automatically consistent or correct. A job may read files while upstream data is still being written, process duplicate inputs or produce partial output after failure. Atomic commit protocols, manifests and table formats with snapshot isolation can reduce these risks.

3.2 Stream Processing

Stream processing operates on an unbounded sequence of records. It is used for fraud detection, operational monitoring, personalised recommendations, industrial control and real-time dashboards. Streaming systems must address continuous input, variable arrival rates, out-of-order records, failures and unbounded state.

A streaming pipeline usually contains a source, transformations, stateful operators, checkpoints and sinks. Backpressure occurs when downstream operators cannot consume data as quickly as upstream operators produce it. A robust system either slows producers, buffers data, scales consumers, sheds low-value load or applies a defined degradation policy.

Streaming results should be described precisely. “Real time” may mean sub-second response, results within several seconds, or simply processing throughout the day rather than once nightly. Service-level objectives should specify latency percentiles, completeness, availability and acceptable error rates.

3.3 Graph Processing

Graph data consists of vertices and edges, possibly with attributes and timestamps. Examples include social networks, transport networks, knowledge graphs, dependency graphs and financial transaction networks.

Graph algorithms often require repeated communication among neighbouring vertices. Two broad approaches are common:

• Graph-centric processing: computation is organised around vertices or edges. In a bulk-synchronous parallel model, each superstep applies vertex computation, exchanges messages and then synchronises.
• Data-centric processing: edges and vertices are represented as records and processed using joins, grouping and iterative dataflow.

Graph-centric systems can reduce unnecessary processing when only a small frontier is active, but synchronisation and skew can be expensive. Data-centric systems integrate well with relational infrastructure but may repeatedly shuffle large edge sets.

For an iterative algorithm, the choice of partitioning is critical. A partition that places connected vertices together reduces network communication, but finding an optimal partition is computationally difficult and dynamic graphs may change the best placement over time. Replication can reduce communication for high-degree vertices, at the cost of additional storage and consistency management.

4. Partitioning, Locality and Skew

Partitioning determines which worker processes each record. Common strategies include:

• Hash partitioning: a hash of a key selects a partition. It is simple and distributes many keys well, but a few very frequent keys can create skew.
• Range partitioning: key intervals are assigned to partitions. It supports ordered scans and range queries but can become unbalanced when values are unevenly distributed.
• Round-robin partitioning: records are distributed cyclically. It balances volume but does not preserve key affinity.
• Consistent hashing: changes in cluster membership move comparatively fewer keys, which is useful in distributed services.
• Graph-aware partitioning: related vertices or edges are placed together to reduce cut edges and communication.

A partitioning function should satisfy the semantic requirement of the operation. A keyed aggregation requires all records for a key to meet at the same logical operator instance. A range query benefits from ordered ranges. A stateless filter may use any balanced partitioning.

Data locality means executing computation near the data it needs. Locality can be storage-local, rack-local or merely within the same region. Locality reduces network traffic and often improves latency, but it should not be pursued at the expense of severe load imbalance. Moving a small computation to data may be cheaper than moving a large dataset to a computation node.

Skew occurs when some partitions receive disproportionately more data or work. It may be caused by popular keys, high-degree graph vertices, uneven file sizes or correlated event bursts. Symptoms include straggler tasks, memory pressure and long tails in latency.

Mitigation strategies include salting hot keys, two-stage aggregation, adaptive repartitioning, splitting large files, replicating frequently accessed data and assigning more resources to heavy partitions. Salting must be used carefully: it changes the grouping key and therefore requires a final aggregation step to reconstruct the original key’s result.

5. Time, Windows and State

Streaming systems must distinguish at least three notions of time:

• Event time: when the event actually occurred.
• Ingestion time: when the system accepted the event.
• Processing time: when an operator processed the event.

Event time is usually the most meaningful for business analysis, but it requires mechanisms for out-of-order and late events. Processing-time windows are simpler but can produce results that depend on machine delay rather than the phenomenon being measured.

A window limits an unbounded computation to a finite subset of records. Common windows include:

• Tumbling windows: non-overlapping intervals, such as 10:00–10:05 and 10:05–10:10.
• Sliding windows: overlapping intervals, such as a five-minute window updated every minute.
• Session windows: periods of activity separated by an inactivity gap.

A watermark is an operational statement about progress in event time. Informally, a watermark of time `t` indicates that the system believes events earlier than `t` are unlikely to arrive, subject to the implementation’s lateness policy. A watermark is not proof that no earlier event exists. Systems may accept late data, update prior results, send corrections or route excessively late data to a side output.

Stateful operators maintain information across records, such as counts, joins, machine-learning features or per-user sessions. State requires a lifecycle policy: how it is partitioned, checkpointed, restored, compacted and eventually expired. Without retention or time-to-live rules, state can grow without bound.

For a keyed count, state may be represented as a mapping from key to count. If events are duplicated, a naïve increment produces an incorrect result. Deduplication may use event identifiers and a bounded state store, but it depends on stable identifiers and a defined retention horizon.

6. Delivery Semantics and Exactly-Once Claims

Delivery semantics are often described as:

• At-most-once: an event may be lost but is not intentionally redelivered.
• At-least-once: an event is redelivered until acknowledgement or progress is recorded; duplicates are possible.
• Effectively-once or exactly-once processing: under specified assumptions, the externally observable result is equivalent to each accepted event affecting the result once.

Exactly-once is not a universal property of a pipeline. It must be specified across boundaries. A processor may guarantee that internal state is recovered consistently, while a non-transactional external database still receives duplicate writes after a failure.

A failure scenario illustrates the issue. Suppose a consumer reads an event, updates its state and then crashes before recording its offset. On restart it reads the event again. If the state update is not atomic with offset progress, the event may be counted twice. A system can address this through transactional state and offset commits, idempotent writes, deterministic recomputation or unique event identifiers.

A credible exactly-once claim should state:

1. Which input events are considered accepted.

2. The failure model, including crash and network-partition assumptions.

3. The scope of atomicity: internal state, output sink or entire business transaction.

4. How retries, duplicates and late events are treated.

5. Whether external side effects are transactional or idempotent.

6. The behaviour during recovery and replay.

In many practical systems, at-least-once delivery combined with idempotent consumers is an effective and understandable design. Idempotence means applying the same operation repeatedly has the same intended effect as applying it once. For example, setting a record to a value identified by version can be idempotent; blindly incrementing a counter is not.

7. Data Quality and Governance

Distributed scale magnifies data-quality problems. A small percentage of invalid records can represent millions of errors, and inconsistent schemas can cause different workers to interpret the same field differently.

Important quality dimensions include:

• Validity: values conform to type, range and format rules.
• Completeness: required records and fields are present.
• Uniqueness: duplicates are controlled or explicitly represented.
• Consistency: related datasets agree on definitions and constraints.
• Timeliness: data arrives within the required operational interval.
• Accuracy: values adequately represent the real-world phenomenon.
• Lineage: the origin and transformations of data are traceable.

Quality controls should be placed at multiple points. At ingestion, validate schema, identifiers, timestamps and security attributes. During processing, monitor null rates, duplicate rates, distribution shifts and referential integrity. At output, test aggregates against reconciliations, known totals or sampled source records.

A useful design separates invalid data from valid data rather than silently discarding it. Quarantine or dead-letter streams should preserve the original record, validation error, processing time and pipeline version. This supports investigation and controlled reprocessing.

Governance includes data ownership, retention, access control, cataloguing, auditability and definition of business metrics. A “customer,” “active user” or “fraudulent transaction” must have an agreed meaning; otherwise technically correct pipelines may produce conflicting reports.

Privacy must be considered throughout the pipeline. Data minimisation, pseudonymisation, encryption, access logging and retention limits reduce exposure. Sensitive attributes should not be copied into every intermediate dataset merely because they are available. Automated analytics can also reproduce historical bias, especially when training data reflects unequal measurement or enforcement.

8. Designing Real-Time Analytics Pipelines

A practical architecture often combines an event log, stream processors, durable analytical storage and serving systems. Events may be retained in raw form, transformed into curated tables and aggregated into low-latency views. A batch process can periodically reconcile streaming results with historical data.

The following design sequence is useful:

1. Define the business decision and its required freshness.

2. Specify event schemas, identifiers, ordering and retention.

3. Identify whether calculations use event time or processing time.

4. Select partition keys that support both parallelism and required joins.

5. Define windows, lateness allowances and correction behaviour.

6. Identify stateful operators and state-retention limits.

7. Decide how failures, replay and duplicates are handled.

8. Select sink semantics and make side effects idempotent where possible.

9. Establish quality metrics, lineage and access controls.

10. Test using historical replay, synthetic disorder, duplicates and failure injection.

A real-time dashboard should not be judged only by average latency. Percentile latency, freshness, missing-event rate, correction frequency and resource cost are often more informative. A dashboard that is fast but systematically excludes late events may be less useful than one that is slightly slower but complete and auditable.

9. Worked Example: Windowed Fraud Indicator

Assume a payment stream contains events with fields `(transactionId, cardId, amount, eventTime)`. The requirement is to flag a card when at least three transactions occur within any ten-minute event-time window and the total amount exceeds 1,000 monetary units.

A suitable pipeline is:

1. Validate the identifier, amount and timestamp.

2. Deduplicate using `transactionId` within a defined retention period.

3. Partition by `cardId` so that all events for a card reach the same keyed operator.

4. Assign events to ten-minute sliding windows, for example with a one-minute slide.

5. Maintain per-window count and amount as state.

6. Use a watermark and allow a specified lateness interval.

7. Emit an alert when both conditions are satisfied.

8. Store the alert with the contributing transaction identifiers and pipeline version.

Consider three valid transactions for one card at event times 12:01, 12:06 and 12:09, with amounts 420, 350 and 310. The count is 3 and the total is 1,080, so the condition is satisfied for a window containing all three events. If the 12:06 event arrives after the watermark has passed, the system must follow its late-data policy: update the alert, emit a correction, or place the event in a late-data stream.

This example demonstrates why a simple count query is insufficient. Correctness depends on identity, event time, partitioning, state retention, lateness and output semantics.

10. Illustrative Case Study: Urban Transit Operations

Background

Consider an illustrative metropolitan transit authority operating buses and trains. Vehicles emit location, speed, passenger-count and fault events. The authority wants to detect service disruption quickly, estimate crowding and produce reliable daily performance reports.

Problem

A nightly batch report is too slow for disruption response. A first streaming prototype produces duplicate alerts during network outages, misses events from vehicles with intermittent connectivity and generates highly unbalanced workloads because a few central stations produce most passenger events.

Analysis

The system contains both bounded and unbounded workloads. Historical performance reporting is naturally batch-oriented, while disruption detection is stream-oriented. Station-based partitioning creates skew, and vehicle clocks are not perfectly synchronised. Therefore, processing time alone would produce misleading arrival intervals. The system also needs replay for investigating incidents.

Approach and Solution

The authority adopts a durable event-stream model. Events receive globally unique identifiers and include vehicle time, ingestion time, schema version and source metadata. Raw events are retained for replay. A streaming dataflow validates records, routes them using a partition key designed to balance load, and maintains event-time windows for station crowding and route delay.

For hot stations, the system uses controlled subpartitioning and a second-stage aggregation. Vehicle and route state is stored in checkpointed keyed state with expiration rules. Watermarks permit a defined period of late vehicle events; later corrections are recorded rather than silently ignored. Alerts are written using an idempotent alert key based on route, condition and time window.

A batch process periodically recomputes daily results from retained raw events. Its output is compared with streaming aggregates, and discrepancies are classified as late data, invalid data, duplicate data or processing defects. Access controls restrict personally identifiable information, and passenger-level data is aggregated before operational dashboards are exposed.

Outcome

The architecture supports rapid operational indicators while preserving the ability to replay and audit events. Duplicate alerts are reduced through stable identifiers and idempotent sinks. Hot-station skew is controlled through two-stage aggregation. Daily reports become more trustworthy because batch recomputation provides an independent reconciliation path.

Learning Points

The case demonstrates that real-time analytics is a systems-governance problem as well as a computation problem. Event schemas, time semantics, partitioning, replay, quality controls and privacy protections are essential parts of the design.

11. Challenges, Limitations and Contemporary Developments

Distributed data processing remains difficult because failures, disorder and partial observability are unavoidable. Network partitions can separate producers, processors and sinks. Autoscaling may alter partition assignments while state is being moved. Schema evolution can break old consumers. Large state stores can make recovery slow even when computation is otherwise fast.

There is also no single ideal processing model. MapReduce offers robust batch recovery but can impose high materialisation cost. Dataflow offers expressive pipelines but exposes complex state and time semantics. Actors provide isolation and natural entity-oriented concurrency but require additional structures for global analytics. Event streams provide replay and decoupling but demand careful schema, retention and offset management.

Contemporary systems increasingly unify batch and stream processing through common APIs, incremental computation, materialised views and lakehouse-style storage. Adaptive query execution can respond to runtime statistics. Approximate algorithms, sketches and sampling can reduce memory and communication for very large datasets, but their error bounds must be reported. Hardware acceleration and specialised vectorised engines improve analytical throughput, while machine learning is increasingly used for anomaly detection, workload prediction and data-quality monitoring.

AI does not remove the need for distributed-systems reasoning. A model-training pipeline still depends on partitioning, reproducibility, feature freshness, label quality and protection against data leakage. Generative systems also require traceability and monitoring when outputs influence decisions.

Best Practices

• Design from explicit latency, completeness, correctness and cost requirements.
• Preserve immutable raw data when replay and audit are important.
• Use stable event identifiers and make external writes idempotent where possible.
• Choose partition keys based on semantic affinity and workload balance.
• Measure and mitigate skew rather than assuming hashing solves it.
• Distinguish event time, ingestion time and processing time.
• Bound state with retention, compaction and expiration policies.
• Treat watermarks as progress estimates, not guarantees of completeness.
• State exactly-once claims with their scope and assumptions.
• Reconcile streaming outputs with independent batch or control totals.
• Quarantine invalid records with diagnostic metadata.
• Version schemas, transformations and metric definitions.
• Test duplicates, out-of-order events, late data, worker failure and replay.
• Apply privacy, access control and retention principles throughout the pipeline.

Chapter Summary

Distributed data processing coordinates computation across machines while managing communication, failure, state and data quality. MapReduce is a robust batch model based on map, shuffle and reduce stages. Dataflow represents pipelines as connected operators and supports both bounded and unbounded computation. Actors organise isolated stateful entities communicating through messages, while event-stream systems treat events as durable, replayable facts.

Batch processing is suitable for historical and deterministic workloads; stream processing supports continuously arriving data but must address time, disorder, state and backpressure. Graph processing requires careful management of neighbour communication, synchronisation and partition quality. Hash, range, round-robin and graph-aware partitioning each have appropriate uses, while skew can create stragglers and must be actively controlled.

Windowing converts unbounded streams into finite computations. Event time, watermarks and lateness policies determine the meaning and completeness of results. Stateful processing requires checkpointing, recovery and bounded retention. Exactly-once is not an absolute guarantee: its validity depends on input acceptance, failure assumptions, transactional boundaries and sink behaviour.

Reliable analytics also depends on validity, completeness, uniqueness, consistency, timeliness, accuracy and lineage. Real-time systems should combine operational responsiveness with replay, reconciliation, governance and privacy protection. The best architecture is therefore selected not only for speed, but for transparent semantics, recoverability, quality and fitness for the decision being supported.

Key Terms

MapReduce: A batch-processing model using map, shuffle and reduce stages.

Dataflow: A directed graph of operators through which data records travel.

Actor model: A concurrency model based on isolated actors communicating by messages.

Event stream: An ordered or append-oriented sequence of durable events.

Partitioning: Dividing data or computation among distributed workers.

Data locality: Executing computation near the data it requires.

Skew: Uneven distribution of data or work across partitions.

Shuffle: Network redistribution of records so equal keys meet.

Batch processing: Processing a finite, identifiable dataset.

Stream processing: Processing continuously arriving, potentially unbounded data.

Event time: The time at which an event occurred in the source domain.

Processing time: The time at which a worker processes an event.

Watermark: An estimate of progress beyond which earlier events are unlikely to arrive.

Window: A bounded logical subset of an unbounded stream.

Stateful operator: An operator that retains information across records.

Idempotence: The property that repeated application has the same intended effect as one application.

At-least-once delivery: Processing semantics that permit redelivery and duplicates.

Exactly-once claim: A scoped assertion that accepted events affect observable results once under stated assumptions.

Data lineage: Traceability of data origins and transformations.

Backpressure: Flow-control behaviour when downstream processing is slower than input production.

Review Questions

Short Answer Questions

1. What are the principal stages of a distributed data-processing pipeline?

2. Why is a combiner in MapReduce considered an optimisation rather than a reducer replacement?

3. Distinguish event time from processing time.

4. What is data skew, and why can it produce straggler tasks?

5. Why must exactly-once claims specify their scope and assumptions?

Long Answer Questions

1. Compare MapReduce and dataflow models with respect to fault recovery, latency, iteration and expressiveness.

2. Explain how partitioning and locality influence the performance of joins and aggregations.

3. Discuss tumbling, sliding and session windows, including the role of watermarks and late data.

4. Analyse the actor and event-stream models for an application containing millions of independent devices.

5. Develop a governance and data-quality strategy for a real-time analytical pipeline.

Analytical / Application Questions

1. Design a streaming pipeline that detects suspicious account activity using event-time windows. Specify partitioning, state, deduplication, lateness and output semantics.

2. A batch job has one very slow reducer while the other reducers finish quickly. Diagnose likely causes and propose technically justified remedies.

3. Evaluate whether an at-least-once pipeline with idempotent sinks can satisfy a business requirement described as “exactly once.” State the assumptions and limitations that must be documented.

Multiple Choice Questions with Answers

Question 1

In MapReduce, what is the primary purpose of the shuffle phase?

A. To compile mapper code

B. To group records with the same key for reducers

C. To encrypt the input dataset

D. To replicate every output to all workers

Correct Answer: B. To group records with the same key for reducers

Question 2

Which processing model most directly represents computation as operators connected by data streams?

A. Dataflow

B. Actor model

C. Round-robin scheduling

D. File replication

Correct Answer: A. Dataflow

Question 3

Which partitioning strategy is generally appropriate when all records for the same key must reach one aggregation operator?

A. Random partitioning

B. Hash partitioning by the key

C. Partitioning by arrival time only

D. Replication to every worker

Correct Answer: B. Hash partitioning by the key

Question 4

What does a watermark primarily represent in event-time stream processing?

A. A proof that no future event exists

B. An estimate of progress in event time

C. The physical location of a worker

D. The number of active partitions

Correct Answer: B. An estimate of progress in event time

Question 5

Which window type consists of non-overlapping fixed intervals?

A. Session window

B. Sliding window

C. Tumbling window

D. Unbounded window

Correct Answer: C. Tumbling window

Question 6

Why can at-least-once processing produce duplicate results?

A. Workers never store state

B. An event may be redelivered after progress was not recorded

C. Hash partitioning always loses records

D. Watermarks delete every late event

Correct Answer: B. An event may be redelivered after progress was not recorded

Question 7

Which is an effective technique for mitigating a hot aggregation key?

A. Salting followed by a second-stage aggregation

B. Removing the key from every record

C. Forcing all records onto one worker

D. Disabling fault recovery

Correct Answer: A. Salting followed by a second-stage aggregation

Question 8

Which data-quality dimension concerns whether required records or fields are present?

A. Completeness

B. Uniqueness

C. Latency

D. Locality

Correct Answer: A. Completeness

Question 9

What is a major advantage of durable event streams?

A. They eliminate all ordering problems

B. They permit replay and independent consumers

C. They guarantee zero network failures

D. They remove the need for schemas

Correct Answer: B. They permit replay and independent consumers

Question 10

Which statement best describes a credible exactly-once guarantee?

A. It is automatically true whenever a system uses checkpoints

B. It applies universally to every external side effect

C. It is scoped to stated failure, state and sink assumptions

D. It means no event can ever arrive late

Correct Answer: C. It is scoped to stated failure, state and sink assumptions