Networking for Big Data, Distributed Systems, and Edge Computing

Introduction

Modern data-intensive applications rarely run on a single computer or within one data centre. Search engines, recommendation platforms, financial systems, scientific simulations, autonomous vehicles, industrial monitoring, and generative-AI services distribute computation and storage across clusters, regions, cloud platforms, and edge locations. In these systems, the network is not merely a transport mechanism. It determines application latency, scalability, fault behaviour, consistency, energy consumption, and operational cost.

A distributed data-processing job may move terabytes between storage and compute nodes. A stream-processing system may require millisecond-level delivery of continuously changing events. A content-delivery network (CDN) may serve the same object from thousands of geographically distributed caches. An edge system may exchange small but time-critical messages over unreliable wireless links. These workloads have different communication patterns and therefore require different network designs.

This chapter examines networking principles for big data, distributed systems, parallel computing, content delivery, and edge/fog computing. Particular attention is given to locality, data movement, synchronization, congestion, consistency, reliability, and the trade-off between centralised cloud processing and distributed edge processing.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain the principal communication patterns used in distributed storage and parallel computing.
2. Analyse bandwidth, latency, throughput, and data-movement costs.
3. Distinguish scale-out, partitioning, replication, shuffling, multicast, and content distribution strategies.
4. Evaluate locality-aware scheduling and data-placement policies.
5. Explain synchronization, consistency, fault tolerance, and flow control in distributed systems.
6. Design suitable communication paths for stream-processing and edge/fog applications.
7. Assess the security, governance, and operational implications of distributed data movement.

1. The Network as a Computing Resource

In a conventional application, a processor requests data from memory or storage. In a distributed application, data may reside on another rack, another availability zone, or another continent. The performance of an operation is consequently determined by both computation and communication.

A useful first-order model for transferring a message of size S bytes across a path is:

where T_transfer is the transfer time, L is fixed end-to-end latency in seconds, S is message size in bytes, and B is effective bandwidth in bytes per second. The model assumes a single dominant path, no significant queueing, and a stable effective bandwidth. Real systems also incur protocol processing, serialization, encryption, retransmission, and queueing delays.

For a sequence of messages, the total time is approximately:

where n is the number of messages and D is the total number of transmitted bytes. This relationship explains why many small messages can be slower than one large message carrying the same data: the latency term is paid repeatedly.

Bandwidth and latency are not interchangeable. High bandwidth is valuable for bulk transfers, whereas low latency is essential for interactive services, coordination protocols, and control loops. A high-bandwidth link can still provide poor performance for a workload consisting of millions of small requests.

1.1 Important performance measures

The mean latency alone is often misleading. A service with a 20 ms average but a 2-second p99 may be unacceptable for a user-facing application. Distributed calls amplify tail behaviour because a request may wait for the slowest of several parallel subrequests.

2. Communication Patterns in Distributed Storage

Distributed storage systems divide data among multiple machines to obtain capacity, parallelism, and fault tolerance. The network is used for client access, replication, rebalancing, metadata operations, repair, and background validation.

2.1 Partitioning and sharding

Partitioning divides a logical dataset into subsets. Range partitioning assigns contiguous key intervals to servers; hash partitioning applies a hash function to distribute keys; and directory-based partitioning uses metadata to map records to locations.

Hash partitioning generally balances load well when the hash function is appropriate, but it weakens range-query locality. Range partitioning supports efficient ordered scans but can create hot spots when new records concentrate in one range. Consistent hashing reduces the amount of data moved when nodes join or leave, although virtual nodes and careful load monitoring are often required.

2.2 Replication

Replication stores multiple copies of data. Synchronous replication confirms a write only after specified replicas acknowledge it. It can provide strong durability but introduces inter-node latency. Asynchronous replication returns before all replicas are updated, reducing write latency but creating a window of inconsistency or data loss after failure.

If independent replicas have failure probability p and a service is available when at least one of r replicas is available, a simplified availability estimate is:

This assumes independent failures and identical replica availability. The assumption is often unrealistic because power, software, rack, and regional failures can be correlated. Replica placement across failure domains is therefore more important than merely increasing replica count.

Replication traffic can be substantial. If each logical byte is written to k physical replicas, the storage system must transmit approximately k−1 additional copies beyond the original, subject to protocol and erasure-coding details. Erasure coding can reduce storage overhead but usually requires more coordination and network traffic during reconstruction.

2.3 Metadata and data paths

Many storage systems separate a metadata path from a data path. Metadata identifies object locations, permissions, versions, and partitions; the data path transfers the actual payload. Keeping metadata small and highly available prevents large data transfers from overloading coordination services. A metadata bottleneck can limit the entire system even when storage bandwidth remains unused.

2.4 Rebalancing and repair

When a node fails or a cluster expands, partitions must be rebalanced and replicas repaired. Repair competes with foreground requests for network capacity. A production system should throttle repair, schedule it according to workload and failure risk, and prioritise under-replicated data. Repair policies should also avoid moving many replicas through the same switch or availability zone.

3. Parallel Computing and Collective Communication

Parallel applications divide a computation among workers. Communication may be point-to-point or collective.

• Point-to-point communication transfers data between a sender and a receiver.
• Broadcast sends one value to many workers.
• Reduce combines values using an operation such as sum, minimum, or maximum.
• All-reduce computes a reduction and makes the result available to every worker.
• Scatter distributes different portions of data from one process.
• Gather collects portions at one process.
• All-to-all exchanges distinct data between every pair or logical group of workers.

Collective operations are central to parallel scientific computing and distributed machine learning. Their cost depends on message size, worker count, topology, and algorithm. A tree broadcast, for example, can reduce the number of sequential communication steps compared with sending separately to every worker. Ring-based all-reduce can balance link usage but may require multiple communication rounds.

3.1 MapReduce and shuffle traffic

A MapReduce-style computation consists conceptually of map, shuffle, and reduce stages. Mappers process input partitions and emit intermediate key-value pairs. The shuffle groups equal keys and transfers intermediate data to reducers. The shuffle is often the dominant network phase because it may approach an all-to-all communication pattern.

Combiner functions can reduce traffic by performing partial aggregation near the mapper. Compression reduces bytes but consumes CPU. Partitioners should distribute keys evenly; skewed keys can overload one reducer, creating a straggler that determines job completion time.

A simplified job completion model is:

Here i indexes workers. The maximum appears because a stage often completes only when its slowest worker finishes. The model assumes a barrier between stages and does not fully represent overlap, speculative execution, or pipelining.

3.2 Distributed machine learning

Data-parallel training gives each worker a model replica and a different mini-batch. Workers calculate gradients and use all-reduce to obtain a common update. Model parallelism divides a model across devices and therefore requires frequent activation and parameter transfers. Pipeline parallelism divides layers into stages and uses micro-batches to keep stages busy.

Communication compression, gradient accumulation, topology-aware placement, and overlapping communication with computation can improve performance. However, aggressive compression or delayed synchronization may affect convergence, reproducibility, and model quality.

4. Locality and Data-Movement Optimization

The principle of locality is to place computation near the data it uses. Moving computation is frequently cheaper than moving large datasets, particularly when storage is persistent and reusable.

Locality has several forms:

1. Host locality: computation runs on the same machine as the data.
2. Rack locality: computation and data are in the same rack or top-of-rack domain.
3. Zone locality: resources share an availability zone or low-latency region.
4. Geographic locality: processing is placed near users or data sources.
5. Semantic locality: related records are co-partitioned to reduce joins.

A scheduler may prefer a local task, accept a rack-local task, or choose a remote task when waiting would cost more than transferring the data. Locality should therefore be treated as an optimisation objective rather than an absolute rule.

4.1 Data-movement cost

Suppose a job reads D bytes and a fraction f is remote. If local and remote effective bandwidths are B_l and B_r, respectively, a simplified transfer time is:

This excludes latency, contention, replication, and processing overhead. It is useful for comparing alternatives under controlled assumptions.

A practical optimisation sequence is:

1. Identify the largest data transfers.
2. Determine whether the transfer is necessary or caused by poor partitioning.
3. Push filters and projections toward the data source.
4. Use partition pruning and predicate pushdown.
5. Co-locate frequently joined data when access patterns justify it.
6. Compress data when CPU and latency budgets permit.
7. Use caching for repeatedly accessed, relatively stable objects.
8. Measure tail behaviour rather than relying only on averages.

Locality can conflict with load balance. A local task is not beneficial if it runs on an overloaded node while an idle remote node is available. Effective schedulers use weighted policies involving queue length, transfer size, task priority, and failure-domain constraints.

5. Stream Processing and Continuous Communication

Batch systems process bounded datasets; stream systems process unbounded sequences of events. Streams may originate from sensors, application logs, financial transactions, click events, or network telemetry.

A stream-processing architecture commonly includes producers, brokers or queues, processors, state stores, and sinks. Partitioning allows multiple consumers to process a stream in parallel. Ordering is usually guaranteed only within a partition, not across the entire stream.

5.1 Delivery and processing semantics

• At-most-once: an event may be lost but is not redelivered.
• At-least-once: an event is not intentionally lost, but duplicates may occur.
• Exactly-once effect: the final externally visible result behaves as though each event affected state once, usually through transactions, idempotence, deterministic replay, or deduplication.

“Exactly once” is not a universal property of the network. It is an end-to-end application guarantee whose validity depends on the source, processor, state store, sink, and failure model.

Watermarks estimate the progress of event time and permit systems to close windows despite out-of-order arrival. Late-event policies may update prior results, discard late records, or route them for correction. These choices affect accuracy, latency, and operational complexity.

5.2 Backpressure and flow control

When a downstream operator is slower than its input, buffers fill. Backpressure propagates upstream, limiting production or causing spills and drops. Without flow control, queue growth can produce memory exhaustion and cascading failure.

For a stable long-term queue, the average arrival rate λ should not exceed the average service rate μ. In practice, systems require headroom because rates fluctuate and service times are variable. Capacity planning should consider burst size, buffer limits, consumer lag, and recovery traffic.

6. Synchronization, Consistency, and Coordination

Distributed processes do not share a single perfectly synchronized clock. Message delay, clock drift, process failure, and network partition make coordination difficult.

6.1 Logical and physical time

Physical clocks support timestamps but may disagree. Logical clocks capture causal ordering: if event A influences event B, A precedes B in the logical order. Vector clocks provide richer causality information but require metadata that grows with the number of tracked participants.

Clock synchronisation protocols reduce physical-clock error but do not eliminate it. Applications requiring ordering should state whether they need wall-clock ordering, causal ordering, or total ordering.

6.2 Consistency models

Strong consistency makes an operation appear to occur at a single, well-defined point in a global order. Sequential consistency preserves the order of operations from each process but may not correspond to real time. Causal consistency preserves cause-and-effect relationships. Eventual consistency allows replicas to converge when updates stop, but intermediate reads may differ.

Consistency is a system-wide design choice involving latency, availability, conflict handling, and business requirements. A social-media counter may tolerate temporary divergence; a financial ledger generally requires stronger invariants.

6.3 Consensus and quorum communication

Consensus protocols replicate a log or decision among a group of nodes. They require leader election, proposal dissemination, acknowledgements, and log repair. Network partitions and delayed messages are normal failure conditions rather than rare exceptions.

In quorum systems, a read quorum R and write quorum W are often selected so that:

where N is the number of replicas. This condition ensures that a read and write quorum overlap, assuming the quorum model and failure semantics are correctly defined. It does not by itself guarantee linearizability, because version selection, concurrent writes, clocks, and repair procedures also matter.

7. Content Delivery Networks and Distributed Caching

A CDN places caches near users and serves content from an edge point of presence rather than an origin server. The principal benefit is reduced path length and reduced origin load.

If a cache receives H hits out of Q requests, the cache-hit ratio is:

The origin request count is approximately Q(1−h), ignoring revalidation and cache hierarchies. A high hit ratio is not always sufficient: a small number of very large misses can still consume considerable bandwidth.

CDN design includes cache-key construction, freshness policies, invalidation, negative caching, origin shielding, TLS termination, and protection against cache poisoning. Personalised or confidential responses require careful cache-control rules. A cached object must not be served to an unauthorised user merely because the URL is identical.

Anycast routing, DNS-based steering, or application-level traffic management can direct users to an edge location. Routing decisions should consider latency, capacity, health, jurisdiction, and attack conditions rather than geographic distance alone.

8. Edge and Fog Computing

Edge computing places computation near data producers or consumers. Fog computing is often used for a broader hierarchical arrangement between end devices and central cloud services. Typical layers include devices, local gateways, regional edge sites, and central cloud regions.

Edge processing is attractive when applications require low latency, limited backhaul, local autonomy, privacy, or reduced bandwidth usage. Examples include machine-vision inspection, connected-vehicle coordination, augmented reality, industrial control, and health monitoring.

8.1 Placement decisions

A service-placement decision should consider:

• response-time requirements;
• CPU, memory, storage, and accelerator availability;
• energy constraints;
• data sovereignty and privacy;
• mobility and changing network topology;
• reliability and physical security;
• model-update and software-distribution costs.

A simple end-to-end latency decomposition is:

The terms represent sensing, uplink, queueing, processing, downlink, and actuation delays. Edge placement mainly reduces uplink, downlink, and sometimes queueing delay; it cannot remove sensing or processing time. The formula assumes a sequential path and should be extended when stages overlap.

8.2 Intermittent connectivity

Edge nodes may disconnect, restart, or operate with limited bandwidth. Applications should use local buffering, durable queues, idempotent updates, retry budgets, and explicit conflict-resolution policies. Offline operation should not be treated as an exceptional error if disconnection is expected.

8.3 Federated learning and privacy

Federated learning keeps training data on local devices and exchanges model updates. It can reduce raw-data movement, but updates may still reveal information through inference attacks. Secure aggregation, differential privacy, update clipping, encryption, and robust outlier detection may be required. These mechanisms introduce computational, communication, and accuracy costs.

9. Reliability, Congestion, and Failure Domains

Distributed applications should be designed for partial failure. A process may be alive but overloaded; a link may be available but lossy; a route may exist but have unacceptable latency.

Timeouts must be chosen with knowledge of normal latency distributions and retry consequences. An overly short timeout creates duplicate work and retry storms. An overly long timeout delays failure detection. Exponential backoff with jitter reduces synchronised retries. Circuit breakers prevent repeated calls to an unhealthy dependency.

Traffic engineering should account for oversubscription, elephant flows, incast, bufferbloat, and microbursts. Incast occurs when many senders transmit simultaneously to one receiver, potentially filling a switch buffer even when average utilisation is moderate. Application-aware protocols, admission control, prioritisation, and deadline-aware scheduling can mitigate these conditions.

Failure domains should be explicit: process, host, rack, zone, region, provider, and geographic jurisdiction. A replica set whose members share one power domain does not provide meaningful independence against that domain's failure.

10. Security, Governance, and Ethical Considerations

Data movement expands the attack surface. Controls should include mutual authentication, encryption in transit, key rotation, least-privilege access, network segmentation, secure boot for edge devices, vulnerability management, and auditable policy enforcement.

Distributed systems also create governance challenges. Data may cross national borders, retention periods may differ, and replicated copies may outlive deletion requests. Architects must know where data, backups, logs, caches, and derived features reside. Data minimisation and purpose limitation are especially important at the edge, where sensors may collect information about people who are not direct users.

Monitoring should avoid exposing sensitive payloads. Trace identifiers, metadata, and sampled logs can support diagnosis, but observability systems themselves require access control and retention policies. Automated placement and routing may affect service quality across regions or populations; fairness and accountability should therefore be considered in operational policies.

11. Worked Example: Choosing a Processing Location

A video-analytics application receives 120 MB/s from cameras. A central cloud path has an effective uplink capacity of 200 MB/s and a round-trip latency of 45 ms. A local edge site can process the stream and transmit only 8 MB/s of extracted events to the cloud. Assume the edge processor adds 12 ms of processing latency and the cloud processing path adds 80 ms after data arrival.

If raw video is sent centrally, the link has utilisation:

Thus the average offered load is 60%, but bursts and other tenants may make this unsafe. The central path transfers 120 MB every second and exposes the application to WAN congestion.

If the edge site processes the video, the cloud receives only 8 MB/s, a reduction of:

The edge path adds 12 ms locally, while cloud transmission concerns only compact events. If the application requires immediate detection, edge processing is preferable. If forensic analysis requires raw video, a policy may retain selected clips locally and upload them asynchronously under a bandwidth budget. The decision is therefore not simply “edge versus cloud”; it may be a tiered design with real-time edge decisions and delayed cloud aggregation.

12. Applied Case: Illustrative Smart-Manufacturing System

Background

A manufacturing company operates 400 inspection cameras and vibration sensors across four factories. Images are high volume, while vibration events require rapid anomaly detection. The company initially sends all data to a regional cloud platform.

Problem

The system experiences high WAN utilisation, variable alert latency, duplicate alerts during retries, and poor operation during temporary site disconnections. Central storage costs are also increasing.

Analysis

Camera frames are not equally valuable: most are normal and need not be transmitted continuously. Vibration signals require local windowing and feature extraction. Factory networks have different capacities, and the factories are separate failure domains. The application requires immediate local alarms but benefits from long-term cloud-based model training.

Approach and solution

The redesigned architecture uses an industrial edge cluster at each factory. Cameras perform adaptive sampling and send selected frames to local inference services. Vibration sensors publish partitioned events to a local broker. Edge processors maintain a durable local state store and issue idempotent alarm identifiers. Only anomalies, compressed summaries, and selected evidence clips are forwarded to the cloud.

A store-and-forward mechanism retains unsent data during outages. Cloud-to-edge model updates are signed, versioned, and staged. The cloud performs fleet-wide analysis, while factory-level operation continues when disconnected. Traffic classes separate control alarms from bulk evidence uploads.

Outcome

The architecture reduces WAN traffic, gives predictable local alert latency, and limits the effect of cloud or WAN outages. It introduces new responsibilities: edge hardware maintenance, secure local deployment, model-drift monitoring, and synchronisation of locally generated events.

Learning points

1. Data reduction at the source can be more effective than increasing network capacity.
2. Local autonomy must be designed together with reconciliation and governance.
3. Reliability depends on durable queues and idempotent processing, not merely on retries.
4. Edge systems shift complexity from the network core to distributed operations.

13. Contemporary Developments and Best Practices

Modern data-centre networks increasingly use high-speed fabrics, programmable switches, RDMA-capable transports, software-defined networking, and workload-aware telemetry. These technologies can reduce CPU overhead or improve control, but they require careful congestion control, hardware compatibility, and failure handling.

Service meshes provide identity, routing, retries, and observability for service-to-service calls, but they add processing hops and can obscure network behaviour. eBPF-based observability can expose kernel and application events with low intrusion, yet collected data must be governed securely.

AI workloads are driving specialised accelerator networks, collective-communication optimisations, and topology-aware scheduling. Large model serving also motivates tensor parallelism, caching, request batching, and geographically distributed inference. These approaches must be evaluated against model freshness, privacy, cost, and tail latency.

Recommended practices are:

• measure p50, p95, and p99 latency, not only averages;
• separate control, interactive, and bulk traffic;
• place replicas across genuine failure domains;
• make retries bounded, idempotent, and jittered;
• compress or aggregate data at the point where it is generated;
• preserve locality while preventing hot spots;
• design for replay, repair, and partial disconnection;
• monitor queue depth, consumer lag, retransmission, and flow completion time;
• test overload, partition, corruption, and regional-failure scenarios;
• document data residency, retention, and deletion behaviour;
• treat security and governance as architectural requirements.

Challenges and Limitations

No single network architecture optimises all objectives. Centralisation simplifies management and enables global analytics but may increase latency, backhaul traffic, and dependence on WAN connectivity. Edge deployment reduces response time and data movement but increases fleet-management complexity and physical attack exposure.

Locality can conflict with fairness, load balance, and resilience. Replication improves availability but consumes bandwidth and creates consistency problems. Compression saves network capacity but costs CPU and may increase latency. Strong consistency simplifies application reasoning but can reduce availability during partitions. Asynchronous processing improves throughput but complicates user-visible correctness.

Analytical models such as T = L + S/B are useful abstractions, not complete performance predictions. Queueing, contention, protocol behaviour, correlated failures, and workload variability must be measured in realistic experiments. Graduate-level engineering requires both modelling and empirical validation.

Chapter Summary

Big-data and distributed applications depend on communication patterns as much as on processors and storage. Partitioning, replication, repair, shuffling, collective communication, caching, and streaming each impose distinct network demands. Latency dominates small-message interactions, while bandwidth dominates bulk movement; tail latency and queueing often determine real user experience.

Locality-aware placement reduces data movement, but must be balanced against load distribution and fault-domain diversity. Stream systems require partitioning, ordering policies, delivery semantics, backpressure, and late-event handling. Distributed coordination requires explicit consistency models, causal assumptions, quorum rules, and failure handling.

CDNs reduce origin load through geographically distributed caching. Edge and fog systems reduce response time and backhaul traffic by processing data near its source, but introduce operational, security, and governance challenges. Effective designs combine local autonomy, durable buffering, idempotence, secure synchronisation, and cloud-scale aggregation. The central engineering task is to place data, computation, and communication where their combined latency, cost, reliability, privacy, and governance requirements are satisfied.

Key Terms

Latency: Time required for a message or operation to obtain a response.

Throughput: Amount of useful data processed or delivered per unit time.

Goodput: Application data delivered after excluding protocol overhead and retransmissions.

Partitioning: Dividing a dataset or stream into independently managed subsets.

Sharding: Partitioning a dataset across storage or service nodes.

Replication: Maintaining multiple copies of data or state.

Failure domain: A set of components exposed to a common failure cause.

Shuffle: Redistribution of intermediate data among parallel workers by key or partition.

Collective communication: Coordinated communication involving a group of processes.

Locality: Placing computation near the data, users, or devices involved.

Backpressure: Feedback that limits an upstream producer when downstream capacity is insufficient.

Watermark: An estimate of event-time progress used to manage out-of-order streams.

Idempotence: The property that repeating an operation produces the same intended effect.

Quorum: A sufficient subset of replicas required to approve or answer an operation.

Consistency model: Rules describing how updates become visible across replicas.

Content-delivery network: A distributed cache and delivery system located near users.

Edge computing: Processing performed near data sources or end users.

Fog computing: A hierarchical distributed-computing arrangement between devices and cloud.

Tail latency: A high-percentile latency, such as p95 or p99.

Data sovereignty: Legal and policy control over where data is stored and processed.

Review Questions

Short Answer Questions

1. Why can a high-bandwidth network still perform poorly for a workload containing many small messages?
2. Distinguish synchronous replication from asynchronous replication.
3. What is the purpose of backpressure in a stream-processing system?
4. Define locality and identify two forms of locality.
5. Why is exactly-once processing considered an end-to-end property?

Long Answer Questions

1. Explain the communication phases of a MapReduce computation and discuss techniques for reducing shuffle traffic.
2. Compare strong, causal, and eventual consistency in terms of latency, availability, and application complexity.
3. Describe how a CDN reduces origin load and analyse the security risks of caching personalised content.
4. Discuss the architectural advantages and operational limitations of edge computing compared with centralised cloud processing.
5. Explain how failure domains, replication, quorum communication, and repair traffic interact in distributed storage.

Analytical / Application Questions

1. A distributed job sends 2,000 messages, each 64 KB, over a path with 5 ms fixed latency and 800 MB/s effective bandwidth. Estimate transfer time using T ≈ nL + D/B, assuming decimal MB and ignoring protocol overhead. Explain why batching could improve performance.
2. An event stream produces 75,000 events per second, while its consumer processes 60,000 events per second. Analyse the likely effect on consumer lag and propose two architectural remedies.
3. Design a cloud–edge architecture for a remote health-monitoring system that must continue operating during a two-hour WAN outage. Specify data placement, buffering, synchronisation, security, and conflict-resolution policies.

Multiple Choice Questions with Answers

Question 1

Which communication pattern is most characteristic of a MapReduce shuffle?

A. One-to-one transfer from a client to a single server

B. All-to-all redistribution of intermediate records by key

C. Broadcast of one unchanged object to every user

D. Local computation with no inter-worker communication

Correct Answer: B. All-to-all redistribution of intermediate records by key

Question 2

A transfer path has effective bandwidth 500 MB/s and carries 100 MB after a fixed latency of 20 ms. Ignoring queueing and overhead, what is the approximate transfer time?

A. 20 ms

B. 120 ms

C. 200 ms

D. 520 ms

Correct Answer: B. 120 ms

Question 3

Which technique directly reduces network traffic before a MapReduce shuffle?

A. Increasing the number of reducers without changing data

B. Using a combiner for partial aggregation

C. Disabling partitioning

D. Replacing all data with metadata

Correct Answer: B. Using a combiner for partial aggregation

Question 4

What is the principal purpose of backpressure?

A. To increase event production when consumers slow down

B. To prevent downstream overload by regulating upstream production

C. To guarantee global ordering across all partitions

D. To eliminate the need for durable storage

Correct Answer: B. To prevent downstream overload by regulating upstream production

Question 5

Which statement about exactly-once effects is most accurate?

A. They are guaranteed automatically by a reliable transport protocol

B. They require an end-to-end design involving processing and state or sink handling

C. They mean that packets are never retransmitted

D. They apply only to batch systems

Correct Answer: B. They require an end-to-end design involving processing and state or sink handling

Question 6

If a cache receives 9,000 hits out of 10,000 requests, what is its cache-hit ratio?

A. 0.09

B. 0.50

C. 0.90

D. 9.00

Correct Answer: C. 0.90

Question 7

Which placement decision best exploits edge computing?

A. Sending all raw sensor data to a distant cloud before any processing

B. Performing latency-sensitive inference near the sensor and uploading summaries

C. Disabling local buffering during WAN outages

D. Replicating every packet to every regional site

Correct Answer: B. Performing latency-sensitive inference near the sensor and uploading summaries

Question 8

For N replicas, which simplified quorum condition ensures that a read quorum and write quorum overlap?

A. R + W < N

B. R + W = 0

C. R + W > N

D. R = W = 1 regardless of N

Correct Answer: C. R + W > N

Question 9

Which is a major risk of placing all replicas in one rack?

A. Improved geographic diversity

B. Correlated failure of the rack can remove all copies

C. Elimination of repair traffic

D. Guaranteed lower tail latency under all workloads

Correct Answer: B. Correlated failure of the rack can remove all copies

Question 10

A stream produces 75,000 events/s and a consumer processes 60,000 events/s. If the rates remain constant, what happens to the backlog?

A. It grows by 15,000 events/s

B. It decreases by 15,000 events/s

C. It remains exactly zero

D. It becomes unrelated to processing capacity

Correct Answer: A. It grows by 15,000 events/s