Distributed Systems and Cloud Design Patterns

Introduction

Cloud applications are distributed systems: their components execute on different machines, communicate through networks, and may fail independently. This separation creates scalability and resilience, but also introduces latency, partial failure, concurrency, duplicated messages, stale data, and uncertain ordering. Advanced cloud architecture therefore depends less on selecting individual services and more on composing reliable patterns.

A well-designed cloud system separates concerns, limits the effects of failure, and makes important behaviour explicit. Stateless services simplify horizontal scaling; partitioning distributes data and workload; replication improves availability; caching reduces latency; queues and events decouple producers from consumers; and idempotency makes retries safe. These techniques must be considered together because improving one property can weaken another. For example, aggressive caching improves performance but may reduce freshness, while asynchronous messaging improves responsiveness but introduces eventual consistency.

This chapter develops the principal distributed-systems design patterns required to design, evaluate, and operate cloud-native solutions.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain why distributed systems experience partial failure, latency, and consistency challenges.
2. Apply stateless design, loose coupling, partitioning, replication, and caching.
3. Select queues and event-driven architecture for suitable workloads.
4. Design idempotent operations and safe retry behaviour.
5. Compare consistency models and analyse CAP trade-offs.
6. Apply timeouts, retries, circuit breakers, bulkheads, and sagas.
7. Evaluate the operational, security, governance, and ethical implications of distributed architectures.

1. Distributed Systems as the Foundation of Cloud Architecture

A distributed system consists of independent computing components that communicate over a network and appear to users as one system. Cloud platforms make distribution easy: a service can run across multiple availability zones, a database can replicate data, and a queue can buffer work. However, the network is not a reliable shared-memory bus. Messages can be delayed, duplicated, reordered, or lost; nodes can be unavailable; and clocks may disagree.

1.1 Partial failure

In a centralised application, a process may fail visibly. In a distributed system, one component may fail while others continue operating. A payment service might be unavailable while the shopping cart remains responsive. This is a partial failure. The system must decide whether to fail fast, degrade functionality, queue work, or return an uncertain result.

A timeout is not proof that an operation failed. It may have completed at the remote service but its response may have been lost. This observation explains why retries can accidentally create duplicate payments or orders unless operations are idempotent.

1.2 Latency and availability

Network calls are slower and less predictable than local function calls. A request that depends on six sequential services accumulates latency and exposes more failure points. Architects should therefore minimise synchronous call chains, set explicit timeouts, and use asynchronous communication where immediate completion is unnecessary.

Availability is commonly expressed as the proportion of time a service is usable:

Uptime and downtime must use the same time unit. If a service is available for 99.9% of a 30-day month, the permitted unavailable time is approximately:

This calculation assumes a 30-day month and treats all downtime equally. Real service-level objectives may distinguish planned maintenance, regional failures, and degraded performance.

2. Stateless Design and Horizontal Scalability

A stateless service does not rely on local process memory to preserve client session state between requests. Any instance can handle any request. State is stored in durable, shared systems such as a database, object store, distributed cache, or client-held token.

Statelessness is particularly valuable in auto-scaling environments. When demand increases, additional instances can be launched without transferring sessions from existing instances. When an instance fails, another instance can continue processing requests.

A typical stateless request flow is:

Client → Load balancer → Any application instance → Shared data service

By contrast, a stateful instance may retain a user session only in local memory. This creates a dependency on session affinity, often called sticky sessions. Sticky sessions can be useful temporarily, but they reduce flexibility: a failed instance may lose sessions, and uneven traffic may produce an imbalanced workload.

Statelessness does not mean that the application has no state. Business state still exists; it is simply externalised and managed explicitly. For example, a shopping cart may be stored in a database, while a short-lived session identifier is stored in a secure, encrypted cookie.

Design guidance

• Keep instances replaceable and disposable.
• Store durable state in a managed persistence service.
• Avoid writing important data only to local disk.
• Use externalised configuration and secrets management.
• Make authentication tokens verifiable by every instance.
• Treat local caches as performance optimisations, not the source of truth.

Stateless services can increase database pressure. This is a trade-off, not a reason to reintroduce hidden state. Read replicas, caching, partitioning, and suitable data models can address the pressure.

3. Loose Coupling and Service Boundaries

Loose coupling means that components depend on stable contracts rather than on internal implementation details. A loosely coupled service can change, scale, or fail without forcing simultaneous changes elsewhere.

Important techniques include:

• Contract-based interfaces: Define request formats, responses, error codes, authentication requirements, and versioning rules.
• Asynchronous messaging: Allow a producer to continue without waiting for a consumer.
• Dependency inversion: Depend on an interface or event contract rather than a concrete implementation.
• Schema evolution: Add optional fields before removing or renaming existing fields.
• Bounded contexts: Keep business rules and data ownership within coherent domains.

Service decomposition should not be based only on the number of classes or database tables. A service boundary should reflect business ownership, security boundaries, scaling characteristics, and transaction requirements. Excessive decomposition creates distributed transactions and operational complexity. A modular monolith can be more appropriate when the domain is small or the team lacks operational maturity.

4. Partitioning and Replication

4.1 Partitioning

Partitioning, or sharding, divides data or work across multiple nodes. A partition key determines where a record is placed. Good partition keys distribute load evenly and support common access patterns.

Common approaches include:

• Hash partitioning: A hash of the key selects a partition. It usually balances load but makes range queries difficult.
• Range partitioning: Key ranges are assigned to partitions. It supports ordered queries but can create hotspots when new records cluster in one range.
• Geographic partitioning: Data is placed near a region or legal jurisdiction. It can reduce latency and support data-residency requirements.
• Tenant partitioning: Customers are separated by tenant identifier. Large tenants may require dedicated partitions.

A partition key should have high cardinality, distribute traffic, and appear in frequent queries. A monotonically increasing key can overload the newest partition. Consistent hashing can reduce data movement when nodes are added, although rebalancing still requires careful capacity and failure planning.

Cross-partition operations are expensive because they require coordination. Architects should design APIs around partition-local operations where possible and use asynchronous aggregation for global views.

4.2 Replication

Replication maintains multiple copies of data or service instances. It improves availability and read capacity, but introduces replica lag and conflict-management problems.

In synchronous replication, an update is acknowledged only after selected replicas confirm it. This can improve durability and consistency but increases write latency and may reduce availability during network partitions. In asynchronous replication, the primary acknowledges before replicas have applied the update. This improves responsiveness but creates a window in which a replica may be stale.

Replication strategies include:

• Leader–follower: One leader accepts writes; followers replicate and often serve reads.
• Multi-leader: Multiple nodes accept writes, requiring conflict resolution.
• Leaderless: Clients or coordinators communicate with several replicas, often using quorum rules.

For a simplified quorum system with replication factor N, write quorum W, and read quorum R, read and write sets overlap when:

This condition supports read-after-write visibility under specified assumptions, including reliable quorum coordination and conflict resolution. It does not by itself guarantee global linearizability, because timing, failures, and version selection also matter.

5. Caching for Performance and Resilience

A cache stores a copy of data closer to the consumer. It reduces latency, database load, and sometimes the impact of downstream failure. Typical layers include browser caches, content delivery networks, reverse proxies, application-memory caches, and distributed caches.

The main design questions are:

1. What data may be stale?
2. How long may it remain stale?
3. Who invalidates it?
4. What happens on a cache miss?
5. Can an attacker infer or manipulate cached data?

Common patterns are:

• Cache-aside: The application reads the cache; on a miss it reads the database and populates the cache. Writes usually update the database and invalidate the cache.
• Read-through: The cache itself loads missing data from the source.
• Write-through: Writes update the cache and source together.
• Write-behind: The cache acknowledges writes and persists them later, improving speed but increasing data-loss risk.

Cache-aside is flexible but can produce stale values and a “thundering herd,” where many requests simultaneously reload an expired item. Mitigations include request coalescing, jittered expiry times, prewarming, and rate limiting.

The cache hit ratio is:

If a cache serves 9,000 hits and 1,000 misses, its hit ratio is 0.90, or 90%. A high hit ratio is not automatically beneficial if cached values are incorrect, expensive to generate, or unsafe to expose.

Security controls must address tenant isolation, encryption, access control, sensitive-data retention, and cache poisoning. Personal or regulated data should have carefully defined retention and invalidation rules.

6. Queues and Event-Driven Architecture

A queue separates work submission from work execution. Producers place messages on a queue, and consumers process them later. This absorbs bursts and allows each side to scale independently. A queue is suitable for tasks such as image processing, report generation, email delivery, and order fulfilment.

A queue may provide at-most-once, at-least-once, or effectively-once processing semantics. At-least-once delivery is common because a consumer may fail after processing but before acknowledging. The message is then delivered again. Consumers must therefore be idempotent.

A dead-letter queue stores messages that repeatedly fail. It prevents one malformed or impossible message from blocking normal processing. Operational procedures must inspect, repair, replay, or discard dead-letter messages safely.

Event-driven architecture uses events to announce facts, such as OrderPlaced or PaymentCaptured. An event differs from a command: a command asks a component to perform an action, whereas an event records that something happened. Events can be published through a broker and consumed by multiple independent services.

Benefits include scalability, extensibility, and temporal decoupling. Costs include eventual consistency, difficult debugging, schema evolution, duplicate delivery, and uncertain event ordering. Event contracts should include a unique event identifier, event type, version, timestamp, source, aggregate identifier, and relevant payload.

The transactional outbox pattern addresses the problem of updating a database and publishing an event atomically. The service writes the business change and an outbox record in one local database transaction. A separate publisher reads unsent outbox records and sends events. The publisher may send an event more than once, so consumers still require idempotency.

7. Idempotency and Reliable Retries

An operation is idempotent if repeating it produces the same intended result as performing it once. Mathematically, an operation f is idempotent when:

The equation describes the observable effect, not necessarily the absence of repeated internal processing. An HTTP PUT that sets an account address to a specified value is naturally idempotent; a POST that creates a new order may not be unless an idempotency key is supplied.

An idempotency design commonly includes:

1. The client generates a unique key for the logical operation.
2. The service stores the key with the result and relevant request fingerprint.
3. A repeated key returns the stored result rather than creating a second effect.
4. Conflicting reuse of a key with different request data is rejected.
5. The record is retained long enough to cover realistic retry and recovery periods.

Retries should be limited to transient failures such as connection resets, temporary overload, or service-unavailable responses. They should not repeat validation failures or permission denials. Exponential backoff reduces synchronized retry storms:

Here, d₀ is the initial delay, k is the zero-based retry number, dₘₐₓ is the maximum delay, and jitter is a random variation. Units must be consistent, normally milliseconds or seconds. A retry budget and an overall deadline are essential; otherwise retries amplify an outage.

8. Consistency Models and CAP Trade-offs

Consistency describes what values a read is allowed to observe after writes occur. Important models include:

• Eventual consistency: If updates stop, replicas eventually converge, but reads may temporarily return older values.
• Read-after-write consistency: A client that has successfully written a value can subsequently read that value.
• Monotonic reads: A client does not observe a newer value and later move backward to an older value.
• Strong or linearizable consistency: Each operation appears to occur atomically at a point between invocation and completion, respecting real-time ordering.
• Causal consistency: Causally related operations are observed in an order that preserves their cause-and-effect relationship.

The CAP theorem concerns a system that experiences a network partition. It states that a distributed data system cannot simultaneously guarantee all three of Consistency, Availability, and Partition tolerance in the strongest interpretation. Since network partitions must be tolerated in a distributed deployment, the practical choice during a partition is often between consistency and availability.

CAP is not a statement that a system has only two properties at all times, nor does it mean that consistency is simply “good” and availability is simply “bad.” Architects choose per operation and per business domain. A social-media like may tolerate delayed visibility; an account balance or inventory reservation may require stronger coordination.

A useful design process is:

1. Identify the business invariant, such as “inventory cannot be negative.”
2. Identify the permitted user-visible delay.
3. Define behaviour during partitions and stale reads.
4. Select storage and communication semantics accordingly.
5. Test failure behaviour rather than documenting only the healthy path.

9. Resilience Patterns

9.1 Timeouts

Every remote call should have a deadline. An absent timeout allows blocked resources to accumulate. The timeout should reflect the user or workflow deadline and leave time for response processing. Nested services should not each wait for their full independent timeout, or the end-to-end delay can become unacceptable.

9.2 Circuit breakers

A circuit breaker monitors failures. In the closed state, calls flow normally. After a threshold of failures or timeouts, it opens and fails quickly, protecting both caller and dependency. After a recovery interval, it enters half-open, allowing a limited number of test calls. Success closes the circuit; continued failure opens it again.

Circuit breakers are not substitutes for fixing the dependency. They should expose meaningful fallback behaviour and produce metrics for state changes, rejected calls, and recovery.

9.3 Bulkheads and load shedding

Bulkheads isolate resources so that one failing dependency cannot consume all threads, connections, or memory. Separate connection pools and concurrency limits are common examples. Load shedding rejects lower-priority work when capacity is exhausted, preserving critical functions.

9.4 Rate limiting

Rate limiting protects services from accidental overload and abuse. A token-bucket limiter adds tokens at a configured rate up to a capacity; each request consumes tokens. The capacity controls short bursts, while the refill rate controls sustained traffic. Limits should be applied by appropriate identities, such as tenant, user, API key, or source network.

10. Saga Pattern for Distributed Business Transactions

A traditional database transaction can atomically update several records in one database. Across independent services, a two-phase commit protocol may impose latency, blocking, and operational complexity. A saga divides a business transaction into a sequence of local transactions. Each step has a compensating action if a later step fails.

For an online purchase, steps might be:

1. Reserve inventory.
2. Authorise payment.
3. Arrange shipment.
4. Confirm the order.

If shipment cannot be arranged, compensation might release inventory and void payment authorisation. Compensation is a new business action, not a perfect time reversal. A payment capture may not be reversible immediately; a refund could be required instead.

A saga can be coordinated by:

• Orchestration: A central coordinator directs steps and records progress.
• Choreography: Services react to events and trigger subsequent actions.

Orchestration makes workflow state and failure handling visible. Choreography reduces central coordination but can become difficult to understand as event relationships multiply. Both approaches require durable workflow state, idempotent steps, explicit timeouts, observability, and reconciliation for ambiguous outcomes.

11. Integrated Case Study: Illustrative Cloud Retail Platform

Background

Consider an illustrative retailer, Northstar Market, whose monolithic application becomes slow during promotional campaigns. Customers browse products, place orders, pay by card, and receive delivery notifications.

Problem

The checkout process synchronously calls inventory, payment, delivery, and notification components. A payment timeout sometimes leads customers to retry, creating duplicate orders. Product searches overload the primary database, while a single regional deployment causes outages during infrastructure incidents.

Analysis

The system has several distinct requirements. Product browsing prioritises low latency and can tolerate briefly stale prices. Inventory reservation must protect the invariant that stock is not oversold. Payment requests require idempotency and careful auditability. Notifications do not need to block checkout. The architecture therefore should not use one consistency model or one interaction style for every operation.

Approach and solution

The platform introduces stateless API instances behind a load balancer. Authentication state is represented by short-lived signed tokens, while carts are stored in a durable cart service. Product data uses a read-optimised store and cache-aside strategy with controlled expiry.

Orders are partitioned by customer or order identifier to distribute load. Inventory uses a strongly coordinated reservation operation within its ownership boundary. The checkout service creates an order and an outbox record in one local transaction. An event publisher emits OrderCreated.

A saga coordinates inventory reservation, payment authorisation, and shipment. Each step carries the order identifier and an idempotency key. Payment retries use exponential backoff with a deadline. A circuit breaker prevents an unavailable payment provider from consuming all checkout resources. On failure, the saga marks the order as requiring attention or executes compensation, such as releasing inventory.

Notification consumers process events asynchronously. They acknowledge a message only after durable processing and record processed event identifiers. Failed messages move to a dead-letter queue for investigation. Dashboards show latency percentiles, queue age, retry counts, circuit state, cache hit ratio, reservation failures, and reconciliation discrepancies.

Outcome

Browsing scales independently from checkout, promotional traffic is absorbed by caches and queues, and payment retries no longer create duplicate charges. The system remains operationally more complex than the original monolith, so the retailer invests in tracing, runbooks, schema governance, and automated failure testing.

Learning points

• Stateless compute supports replacement and horizontal scaling.
• Different workloads require different consistency decisions.
• At-least-once delivery makes idempotent consumers essential.
• A saga provides business-level coordination without pretending that distributed steps are one atomic database transaction.
• Operational visibility is part of the architecture, not an afterthought.

12. Governance, Security, and Professional Practice

Distributed systems enlarge the security boundary. Every service-to-service call needs authentication, authorisation, encryption in transit, and an accountable identity. Secrets should be managed centrally and rotated. Least privilege should apply to service accounts, queues, databases, and observability systems.

Event payloads may be copied into brokers, logs, caches, backups, and analytics platforms. Data minimisation is therefore important: publish only what consumers need, classify sensitive fields, encrypt appropriate stores, and define retention and deletion processes. Privacy regulations may require access, correction, deletion, residency, and audit capabilities that are difficult when data is replicated across regions.

Reliability decisions also have ethical consequences. An automated retry or fraud rule may repeatedly charge or exclude a customer. Systems should provide traceable decisions, human escalation for exceptional cases, and accurate audit records. Accessibility and equitable service degradation should be considered when load shedding is designed.

Observability should combine logs, metrics, and distributed traces. Correlation identifiers connect a user request to downstream calls, queue messages, and saga steps. Monitoring should measure user outcomes, not only infrastructure health. A service may be “up” while orders are silently accumulating in a dead-letter queue.

13. Contemporary Developments and Limitations

Managed serverless functions, container orchestration, globally distributed databases, and service meshes automate parts of distributed-systems operation. They do not eliminate fundamental trade-offs. A serverless function can still process a message twice; a service mesh can still experience a partition; and a globally replicated database can still expose conflict or lag.

AI-assisted operations can identify anomalous latency, predict capacity needs, summarise incidents, and suggest remediation. Such tools must be governed because automated changes can amplify an outage, and telemetry may contain sensitive information. Human approval, audit trails, bounded permissions, and rollback mechanisms remain important.

The principal limitation of patterns is contextual misuse. Caching stale financial data, applying retries to non-idempotent operations, splitting a small domain into dozens of services, or using eventual consistency where a legal or safety invariant requires strong coordination can create serious harm. Patterns are design tools, not universal recipes.

Best-Practice Checklist

• Define service-level objectives and end-to-end deadlines.
• Keep compute instances stateless and replaceable.
• Choose partition keys from real access and ownership patterns.
• Treat caches as disposable copies with explicit freshness rules.
• Assume messages and requests may be duplicated.
• Make handlers idempotent and retain operation records appropriately.
• Use bounded retries with exponential backoff, jitter, and retry budgets.
• Apply timeouts, circuit breakers, bulkheads, and rate limits.
• Select consistency per business invariant, not by fashion.
• Use an outbox for reliable event publication.
• Design compensation and reconciliation for sagas.
• Secure every service boundary and minimise event data.
• Test partitions, duplicates, delays, overload, and recovery.
• Instrument queue age, stale data, failed workflows, and business outcomes.

Chapter Summary

Cloud applications are distributed systems in which network delay, independent failure, duplication, and inconsistent views are normal engineering conditions. Stateless design enables horizontal scaling and replacement, while loose coupling allows services to evolve independently. Partitioning distributes data and workload; replication improves availability and read capacity but introduces lag and conflict concerns. Caching reduces latency but requires explicit freshness, invalidation, security, and stampede controls.

Queues and events provide temporal and operational decoupling, but commonly offer at-least-once delivery, making idempotent producers and consumers essential. Safe retries require timeouts, bounded exponential backoff, jitter, and operation-specific decisions. Consistency models should be selected according to business invariants and user needs. CAP analysis clarifies the consequences of network partitions, while sagas coordinate multi-service workflows through local transactions and compensation. Circuit breakers, bulkheads, rate limits, observability, governance, and security complete the professional architecture.

Key Terms

Distributed system: A system whose components execute independently and communicate over a network.

Partial failure: Failure of one component while other components continue operating.

Stateless service: A service that does not depend on local memory to preserve client state between requests.

Loose coupling: Dependence on stable contracts rather than implementation details.

Partitioning: Dividing data or work across multiple nodes or partitions.

Replication: Maintaining multiple copies of data or service instances.

Replica lag: The delay before a replica reflects an update accepted elsewhere.

Cache-aside: A pattern in which the application loads missing data into a cache.

Queue: A mechanism that buffers work between producers and consumers.

Event: A record that a significant fact has occurred.

Dead-letter queue: A holding area for messages that repeatedly fail processing.

Idempotency: The property that repetition has the same intended effect as one execution.

Exponential backoff: Increasing retry delays, commonly by powers of two.

Circuit breaker: A failure-control mechanism that stops calls to an unhealthy dependency.

Bulkhead: Isolation of resources to limit failure propagation.

Consistency model: Rules describing which values reads may observe.

CAP theorem: A result concerning consistency, availability, and partition tolerance during network partitions.

Saga: A distributed business workflow composed of local transactions and compensating actions.

Transactional outbox: A pattern that stores a business change and pending event in one local transaction.

Observability: The ability to infer system behaviour from logs, metrics, and traces.

Review Questions

Short Answer Questions

1. Why is a timeout not proof that a remote operation failed?
2. Distinguish statelessness from the absence of application state.
3. What is the purpose of a dead-letter queue?
4. State the quorum-overlap condition using replication factor N, read quorum R, and write quorum W.
5. What are the three principal states of a circuit breaker?

Long Answer Questions

1. Explain how partitioning and replication affect scalability, availability, latency, and consistency.
2. Compare cache-aside, write-through, and write-behind caching, including their risks.
3. Explain why at-least-once message delivery requires idempotent consumers.
4. Discuss how CAP trade-offs should be analysed using business invariants.
5. Compare orchestration and choreography in saga implementations.

Analytical / Application Questions

1. Design a checkout workflow that remains safe when payment responses are delayed, duplicated, or lost. Identify idempotency records, retry rules, and compensation actions.
2. A product catalogue receives heavy read traffic, while price updates occur every few minutes. Propose a partitioning, replication, caching, and consistency strategy, and justify the permitted staleness.
3. A queue’s consumer service is overloaded and its age of oldest message is increasing. Analyse possible causes and propose controls involving concurrency limits, autoscaling, backpressure, dead-letter handling, and observability.

Multiple Choice Questions with Answers

Question 1

Which property allows any healthy application instance to process a request without relying on a previous instance’s local memory?

A. Statelessness

B. Sticky sessions

C. Leader election

D. Write-behind caching

Correct Answer: A. Statelessness

Question 2

A service uses replication factor N = 3, read quorum R = 2, and write quorum W = 2. What is true under the stated quorum assumptions?

A. R + W > N, so read and write sets overlap

B. R + W < N, so overlap is impossible

C. The system has no replication

D. Every operation is automatically globally linearizable

Correct Answer: A. R + W > N, so read and write sets overlap

Question 3

Which pattern stores a business change and an event record in the same local database transaction?

A. Circuit breaker

B. Transactional outbox

C. Bulkhead

D. Cache-aside

Correct Answer: B. Transactional outbox

Question 4

Why are idempotent consumers important with at-least-once delivery?

A. Messages are guaranteed to arrive in order

B. A message may be delivered again after uncertain acknowledgement

C. They eliminate the need for durable storage

D. They guarantee zero network latency

Correct Answer: B. A message may be delivered again after uncertain acknowledgement

Question 5

What is the primary purpose of a circuit breaker’s open state?

A. To route every request to a second database

B. To fail quickly and protect an unhealthy dependency

C. To increase message duplication

D. To disable authentication temporarily

Correct Answer: B. To fail quickly and protect an unhealthy dependency

Question 6

Which consistency model permits temporary stale reads but converges if updates stop?

A. Eventual consistency

B. Linearizability

C. Serial execution only

D. Strict two-phase locking

Correct Answer: A. Eventual consistency

Question 7

Which retry strategy best reduces synchronised retry storms?

A. Immediate unlimited retries

B. Fixed retries with no timeout

C. Exponential backoff with jitter and a retry limit

D. Retrying authorization failures indefinitely

Correct Answer: C. Exponential backoff with jitter and a retry limit

Question 8

In a saga, what is a compensating action?

A. A network packet retransmission

B. A business action that addresses the effect of a completed step

C. A cache read performed before a write

D. A replacement for authentication

Correct Answer: B. A business action that addresses the effect of a completed step

Question 9

Which partitioning approach generally supports ordered range queries but may create hotspots for sequential keys?

A. Range partitioning

B. Random encryption

C. Leaderless replication

D. Token-bucket partitioning

Correct Answer: A. Range partitioning

Question 10

What is a principal risk of write-behind caching?

A. The cache can never serve reads

B. Acknowledged data may be lost before delayed persistence

C. It always requires strong global consistency

D. It prevents horizontal scaling

Correct Answer: B. Acknowledged data may be lost before delayed persistence