6. Service Communication and Event-Driven Architecture

Introduction and Context

A microservices system is not merely a collection of independently deployable applications. Its effectiveness depends on how those services collaborate. A service may need to request information from another service, notify other services that something has happened, coordinate a long-running business process, or process a continuous stream of events. The communication model selected for each interaction influences latency, availability, consistency, scalability, observability, and operational complexity.

In a tightly coupled monolithic application, communication commonly occurs through in-process method calls and a shared database. Microservices replace many of these assumptions with network communication and separately managed data stores. A network call can fail, be delayed, be duplicated, or return an incomplete response. Consequently, service communication must be designed as a distributed-systems problem rather than treated as a remote version of a local function call.

This chapter examines synchronous and asynchronous communication, message brokers, queues, publish-subscribe systems, event streaming, choreography, and orchestration. It also addresses ordering, duplicate delivery, idempotent consumers, dead-letter handling, schema evolution, governance, and the practical trade-offs involved in selecting a communication architecture.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish synchronous request-response communication from asynchronous messaging and event streaming.

2. Select suitable communication mechanisms for different business and technical requirements.

3. Explain the roles of brokers, queues, topics, subscriptions, consumers, and event logs.

4. Design reliable consumers that tolerate retries, duplicate delivery, and partial failure.

5. Analyse ordering, delivery guarantees, dead-letter processing, and schema evolution.

6. Compare choreography and orchestration in distributed business workflows.

7. Apply communication patterns in a microservices case and evaluate their risks.

8. Establish professional practices for observability, security, governance, and operational control.

Communication as a Distributed-System Design Decision

A service interaction has at least two dimensions: the form of communication and the degree of temporal coupling. In synchronous communication, the caller waits for a response. In asynchronous communication, the sender transfers a message or event and continues without waiting for the recipient to complete processing. Temporal decoupling can improve resilience, but it also introduces eventual consistency, delayed feedback, and more demanding operational diagnostics.

The meaning of a message is equally important. A command asks a particular service to perform an action, such as `ReserveInventory`. An event records a fact that has already occurred, such as `InventoryReserved`. A query requests information without intending to change state. Confusing commands and events often produces unclear ownership and accidental coupling. Commands are directed and normally have one logical handler; events are published for interested consumers and may have many handlers.

Communication should therefore be designed around business boundaries. A service should own its data and expose capabilities through explicit contracts. Sharing a database table between services may appear efficient, but it bypasses service boundaries, creates hidden dependencies, and makes independent evolution difficult.

Synchronous Request-Response

Synchronous communication is appropriate when a caller requires an immediate result to continue its operation. HTTP APIs and gRPC are common implementations. For example, a checkout service may synchronously request a product catalogue price when it must validate a request before creating an order.

Its principal advantage is straightforward control flow. Errors can be returned directly, validation is immediate, and a client can provide a clear response to a user. Synchronous calls are often suitable for short, bounded operations with a clear response and acceptable latency.

The cost is temporal coupling. If Service A calls Service B, A depends on B being reachable and sufficiently responsive. A chain such as A → B → C → D multiplies failure opportunities and increases latency. A slow dependency can consume connection pools and threads, producing cascading failure. Resilience mechanisms are therefore essential:

• strict connection and response timeouts;
• bounded retries with exponential backoff and jitter;
• circuit breakers that temporarily stop calls to an unhealthy dependency;
• bulkheads that isolate resources used by different operations;
• fallback responses where a degraded result is safe;
• correlation identifiers for tracing a request across services.

Retries must be used carefully. Retrying a read is generally safer than retrying a non-idempotent write. A payment request retried without an idempotency key might charge a customer twice. A timeout does not prove that the remote operation failed; the operation may have completed while its response was lost. This uncertainty is a central reason for using idempotency keys and durable state transitions.

Synchronous communication does not necessarily imply strong consistency across a complete business process. A service can respond successfully after recording its own state while other services update asynchronously. The communication style and consistency model are related but not identical decisions.

Asynchronous Messaging: Queues, Topics, and Brokers

Asynchronous messaging introduces an intermediary, commonly called a message broker. The producer sends a message to the broker, which stores and routes it until one or more consumers process it. The broker may provide persistence, acknowledgements, retry support, routing, access control, and operational metrics.

A queue generally distributes work among competing consumers. If five workers consume from one queue, a particular message is normally delivered to one worker. Queues are useful for load levelling, background jobs, and work that should be processed once by a logical worker group. A queue can absorb temporary bursts, allowing producers and consumers to operate at different rates.

A topic supports publication to multiple subscriptions. Each subscription receives its own logical copy of the messages. This is useful when an `OrderPlaced` event must be consumed independently by billing, fulfilment, analytics, and customer notification services. The publisher need not know which consumers exist, which reduces direct coupling.

The terms vary across products. Some platforms call a topic with consumer groups a stream, while others distinguish queues, exchanges, subjects, partitions, and subscriptions. Architecture should therefore describe the required semantics—fan-out, competing consumers, retention, replay, ordering, and acknowledgement—rather than relying only on vendor terminology.

A broker does not eliminate failure. Messages can be rejected, expire, become unprocessable, or be delivered again. The producer must know whether a message was accepted durably, and the consumer must define when processing is considered complete. Acknowledging before successful processing risks message loss; acknowledging only after processing may cause duplicates when the consumer crashes immediately after the business action but before acknowledgement.

Event Streaming and Log-Based Collaboration

Event streaming treats events as an ordered, durable record that can be read by one or more consumers. A stream is commonly divided into partitions. Ordering is guaranteed within a partition, while different partitions may be processed concurrently. Consumers track an offset or equivalent position, enabling replay, recovery, and construction of new projections.

Streaming is valuable when events have continuing analytical or operational value. Examples include account transactions, sensor readings, click activity, and changes to customer status. A consumer can rebuild a materialised view by replaying retained events, provided the events contain adequate information and the processing logic remains compatible.

Streaming is not automatically superior to a queue. Long retention and replay create storage and governance obligations. Event streams also require careful partition-key selection. If all events use one key, ordering is preserved but parallelism may be poor. If related events are distributed across partitions, throughput improves but cross-entity ordering becomes difficult. The key should reflect the boundary within which ordering is genuinely required, such as `orderId`.

Event-carried state transfer is another design choice. A thin event might state that an order was updated, requiring consumers to call the order service for details. A rich event may contain the relevant state at the time of publication, reducing follow-up calls but increasing payload size and schema-management responsibility. The appropriate choice depends on autonomy, privacy, change frequency, and consistency requirements.

Choreography and Orchestration

Distributed workflows commonly use choreography or orchestration. In choreography, services react to events without a central coordinator. An order service publishes `OrderPlaced`; inventory reacts and publishes `InventoryReserved` or `InventoryRejected`; payment reacts to the relevant event; fulfilment reacts when prerequisites are met. Choreography can preserve service autonomy and avoid a central bottleneck.

However, extensive choreography may make the overall workflow difficult to understand. Business rules become distributed across event handlers, and it may be unclear which service is responsible for progressing or compensating a process. Circular reactions and hidden dependencies are possible.

In orchestration, a workflow coordinator directs participants by sending commands and interpreting replies or events. An order process manager may request inventory reservation, then payment authorization, then shipment creation. The orchestrator provides a visible process model, centralises timeout and compensation logic, and can report workflow status. Its risks include excessive centralisation, coordinator failure, and the temptation to make the orchestrator aware of internal details of every service.

Neither pattern is universally correct. Choreography suits loosely coupled reactions and domains where services can make independent decisions. Orchestration is often preferable for explicit, long-running processes with ordered steps, deadlines, approvals, and compensation. A hybrid architecture is common: an orchestrator controls a business transaction while domain events notify independent consumers such as analytics or search indexing.

Reliability Semantics and Delivery Guarantees

Messaging systems are often described using three delivery labels. At-most-once delivery may lose messages but does not intentionally redeliver them. At-least-once delivery avoids loss under specified conditions but may deliver duplicates. Exactly-once processing is difficult to guarantee across a broker, consumer, database, and external systems; product-level claims usually apply only to a limited boundary.

For most business systems, at-least-once delivery combined with idempotent processing is a realistic and robust design. A consumer should produce the same effective result if it receives the same message more than once. This can be achieved through several techniques:

• storing a unique event or message identifier in a processed-message table;
• using a database uniqueness constraint to reject repeated business operations;
• applying state transitions conditionally, such as updating only when a version matches;
• using idempotency keys supplied by clients for externally initiated requests;
• making set-based updates rather than incrementing blindly;
• designing downstream APIs to accept and recognise request identifiers.

The idempotency record and business update should normally be committed atomically. If a consumer records the message as processed and then crashes before updating business state, the event may be lost from the consumer’s perspective. If it updates business state and crashes before recording the message, the message will be retried; the business operation must then safely recognise the duplicate.

A common implementation is an inbox pattern. The consumer stores the incoming message in an inbox table within the same transaction as the business update. A unique constraint on the message identifier prevents repeated application. An outbox pattern addresses the opposite direction: a service writes its state change and an outgoing event to an outbox table in one transaction. A separate relay publishes the outbox event to the broker. This prevents the dual-write problem in which a database transaction succeeds but event publication fails, or publication succeeds while the transaction rolls back.

Ordering, Concurrency, and Causality

Ordering is meaningful only when the domain requires it. A shipment cancellation may need to follow a shipment creation event, but unrelated orders need not be globally ordered. Global ordering limits scalability and creates unnecessary coordination.

A practical design assigns a partition or routing key such as customer ID, account ID, or order ID. Events with the same key are processed sequentially by a consumer group, while different keys can be processed concurrently. Consumers should still tolerate out-of-order delivery because retries, multiple subscriptions, migrations, and independent services can disrupt assumptions.

Version numbers, timestamps, sequence numbers, and causation identifiers help detect stale or invalid events. A consumer may reject an update with version 7 if version 8 has already been applied. Timestamps alone are unsafe as a universal ordering mechanism because clocks are not perfectly synchronised and event time is not the same as processing time.

Concurrency also affects projections. Two consumers may update related data simultaneously. Optimistic concurrency control, atomic database operations, partition affinity, and conflict-resolution rules are preferable to assuming that arrival order always represents business order.

Failures, Retries, and Dead-Letter Handling

A failed message should not be retried indefinitely. Transient failures such as temporary database unavailability may succeed after a delay; permanent failures such as invalid schema, missing required data, or an impossible state transition will not. Retry policies should classify failures and use bounded attempts with increasing delays. Jitter prevents many consumers from retrying simultaneously after a shared outage.

After repeated failure, a message may be moved to a dead-letter queue or dead-letter topic. Dead-letter handling is an operational process, not a wastebasket. The system should preserve the original payload, headers, error category, stack or diagnostic information, attempt count, timestamps, and correlation identifiers. Operators need tools to inspect, correct, replay, or quarantine messages safely.

Replay must be controlled. Replaying a message can repeat external effects unless consumers are idempotent. A corrected message may be reprocessed after code or data repair, but the organisation should record who initiated the replay, why it occurred, and what scope it covered. Poison messages—messages that consistently fail—should be isolated so that they do not block unrelated work.

A dead-letter queue can conceal serious defects if dashboards show only successful broker delivery. Operational metrics should include consumer lag, processing latency, retry rate, dead-letter volume, acknowledgement failures, age of the oldest message, and the proportion of events successfully processed within a business deadline.

Contracts and Schema Evolution

An event or message is a contract between its producer and consumers. The contract includes its name, purpose, fields, types, semantic meaning, identifiers, versioning policy, and compatibility expectations. Documentation alone is insufficient; automated contract tests and schema validation reduce accidental breaking changes.

Backward-compatible evolution usually adds optional fields while preserving the meaning of existing fields. Consumers should tolerate unknown fields and avoid assuming that every field is always present. Removing or renaming a field, changing units, narrowing a type, or changing the interpretation of a status can break consumers even when the syntax remains valid.

Several strategies are possible. Explicit versioned event types, such as `CustomerAddressChangedV2`, make changes visible but may increase the number of contracts. A schema registry can store schemas, validate messages, and enforce compatibility rules. Content negotiation or event metadata can allow producers and consumers to identify supported versions. In all approaches, migration should be planned: introduce the new representation, update consumers, run both forms if necessary, and retire the old form only after usage has been measured.

Events should be designed for long-term interpretation. Avoid exposing internal database structures, unclear abbreviations, locale-dependent values, or fields whose meaning changes without notice. Include stable identifiers, occurred-at time, producer information, correlation and causation identifiers, and an explicit schema or event type. Sensitive data should not be placed in widely retained streams unless there is a justified retention and access policy.

Applied Case: Event-Driven Order Fulfilment

Background

Consider an illustrative online retailer, Northstar Retail, operating separate services for orders, inventory, payments, shipping, notifications, and analytics. The original implementation used a synchronous chain: the web application called the order service, which called inventory, payment, shipping, and notification services before returning a response.

Situation and Problem

During promotional periods, the chain experienced high latency and cascading failures. A slow payment provider caused order requests to time out. Some customers retried the request, producing duplicate orders. Inventory reservation occasionally succeeded even though the client received an error. The organisation also wanted analytics consumers and a recommendation platform to receive order activity without modifying the order service for every new subscriber.

Analysis

Not every step required an immediate response. The customer needed prompt confirmation that an order had been accepted, but fulfilment could continue after that point. Inventory reservation and payment authorisation were business actions requiring reliable commands and explicit results. Notifications and analytics were independent reactions. The system also needed a visible process status because an order could remain pending while payment or inventory was unavailable.

Approach and Solution

Northstar introduced an order process manager. The order service synchronously validated the request and created an order with status `PENDING`, using a client-supplied idempotency key to prevent duplicate creation. In the same database transaction, it wrote an `OrderPlaced` record to an outbox.

An outbox relay published the event to a broker. The process manager consumed it and sent a `ReserveInventory` command to an inventory queue. Inventory processed the command idempotently using the command identifier and emitted either `InventoryReserved` or `InventoryRejected`. The process manager then sent an `AuthorizePayment` command only when inventory was reserved. Payment used a provider idempotency key because a timeout could hide a completed authorisation.

If payment succeeded, the process manager sent a `CreateShipment` command and updated the order workflow. If payment failed, it sent a `ReleaseInventory` command and marked the order as requiring customer action. Notification and analytics services subscribed independently to domain events. They did not participate in the critical fulfilment path.

Every consumer used an inbox or equivalent deduplication mechanism. Events carried order ID, event ID, occurrence time, correlation ID, causation ID, and schema version. Retries used exponential backoff. Invalid messages went to a dead-letter topic with diagnostic metadata. Dashboards displayed workflow duration, consumer lag, duplicate rate, and dead-letter counts.

Outcome

The customer-facing request became short and predictable because it no longer waited for all downstream actions. Temporary payment-provider failures no longer exhausted web-service resources. Duplicate client requests mapped to the same order through the idempotency key. The process manager made the workflow state visible, while independent subscribers could be added without changing the fulfilment sequence.

Learning Points

The case illustrates that asynchronous communication is not simply a performance technique. It changes the consistency model and requires explicit workflow state, compensation, observability, and failure handling. The outbox prevents lost events, idempotent consumers tolerate redelivery, and orchestration provides clarity where the business process has ordered steps. Non-critical reactions remain decoupled through publication and subscription.

Professional and Governance Considerations

Communication architecture creates organisational as well as technical dependencies. Teams should agree on ownership of event names, schemas, retention, privacy classifications, and service-level objectives. A producer should not silently change the meaning of a field because several unknown consumers may rely on it. A catalogue of contracts, ownership metadata, and deprecation dates supports responsible evolution.

Security applies to both messages and brokers. Use authenticated producers and consumers, authorise access at topic or queue level, encrypt data in transit and at rest, and limit administrative privileges. Message contents may contain personal, financial, or health information. Retention, deletion, access logging, masking, and regulatory obligations must be considered, particularly because immutable event logs may preserve data longer than operational databases.

Audit requirements can favour durable events, but audit events must be distinguished from ordinary integration events. An audit record should preserve who performed an action, what changed, when it changed, and the relevant authority or context. It should not be assumed that a technical delivery log automatically constitutes a legally adequate audit trail.

Common Design Errors and Limitations

Several recurring errors undermine event-driven systems. Treating a broker as a database leads to unbounded retention and unclear ownership. Assuming exactly-once delivery encourages unsafe consumers. Publishing events that merely mirror internal tables exposes implementation details. Using synchronous calls for every interaction recreates a distributed monolith, while using asynchronous events for every interaction makes simple request-response behaviour unnecessarily difficult.

Other mistakes include retrying permanent failures, omitting correlation identifiers, failing to monitor consumer lag, using a single partition for all traffic, relying on timestamps for strict ordering, and placing irreversible external actions before local idempotency is established. A service may also publish an event before its transaction commits, causing consumers to observe a state that later disappears. Transactional outbox or equivalent atomic publication patterns address this risk.

Event-driven architecture does not remove complexity; it relocates complexity from call stacks to contracts, workflows, operations, and data consistency. The design is justified when independent scaling, temporal decoupling, multiple consumers, long-running processes, or replayable history provide meaningful value.

Current Developments and Best Practices

Contemporary platforms increasingly combine request-response APIs, durable messaging, and streaming rather than selecting one universal mechanism. Managed brokers reduce infrastructure work but do not remove the need for capacity planning, access control, schema governance, and incident response. Cloud-native systems commonly use serverless consumers, managed event buses, workflow engines, and distributed tracing.

AI and analytics applications intensify the importance of reliable event pipelines. Feature computation, anomaly detection, fraud analysis, and real-time recommendations depend on accurate event time, replay capability, lineage, and data-quality controls. AI-generated consumers or transformations should still be subject to schema validation, privacy review, testing, and human governance; automation does not make an incorrect event contract safe.

A practical set of best practices is to:

• choose communication style according to latency, coupling, consistency, and failure requirements;
• define commands, events, and queries separately;
• use timeouts and bounded retries for synchronous calls;
• assume at-least-once delivery and design idempotent consumers;
• apply inbox and outbox patterns where atomicity is required;
• partition streams according to the domain’s real ordering boundary;
• make workflow state and compensation explicit;
• validate contracts and evolve schemas compatibly;
• monitor lag, latency, retries, duplicates, and dead letters;
• protect message data through authentication, authorisation, encryption, and retention controls;
• test failure, replay, reordering, duplication, and partial completion—not only the happy path.

Chapter Summary

Service communication is a foundational microservices design decision. Synchronous request-response offers immediate feedback but creates temporal coupling and can propagate failure. Asynchronous queues and topics provide buffering, fan-out, and independent processing, while event streams add durable retention, replay, and scalable consumption. Commands express requested actions; events express facts that have occurred.

Reliable systems assume that messages may be delayed, duplicated, reordered, or rejected. At-least-once delivery paired with idempotent consumers is commonly more practical than attempting universal exactly-once processing. Inbox and outbox patterns address duplicate handling and the dual-write problem. Partitioning, sequence information, and explicit concurrency controls help manage ordering.

Choreography supports autonomous event reactions, whereas orchestration makes long-running workflows visible and controllable. Dead-letter handling, schema governance, observability, security, privacy, and replay procedures are essential operational responsibilities. Effective architecture combines communication styles deliberately and treats contracts and failure behaviour as first-class design concerns.

Key Terms

Command: A message requesting a specific service or component to perform an action.

Event: An immutable record stating that something has occurred.

Message broker: Infrastructure that accepts, stores, routes, and delivers messages between producers and consumers.

Queue: A messaging structure in which work is generally distributed among competing consumers.

Topic: A publication channel from which multiple subscriptions can independently receive messages.

Consumer: A component that reads and processes messages or events.

Event stream: A durable sequence of events that can be consumed, tracked, and replayed.

Partition: A subdivision of a stream used to provide scalability and usually ordering within a key.

At-least-once delivery: A delivery guarantee in which messages are not intentionally lost but may be delivered repeatedly.

Idempotent consumer: A consumer whose repeated processing of the same message produces one effective business result.

Inbox pattern: A pattern that records received messages and business changes atomically to support duplicate detection.

Outbox pattern: A pattern that stores a state change and its outgoing event in one transaction before publication.

Dead-letter queue: A location for messages that cannot be successfully processed after defined handling attempts.

Consumer lag: The distance between the latest available event and the position processed by a consumer.

Choreography: Coordination in which services independently react to events without a central workflow controller.

Orchestration: Coordination in which a central process manager directs services through a workflow.

Compensation: A corrective action that semantically reverses or offsets an earlier completed action.

Schema evolution: The controlled modification of message structure and meaning over time.

Correlation identifier: An identifier used to associate messages and operations belonging to one broader request or workflow.

Eventual consistency: A model in which distributed replicas or services may temporarily differ but converge after successful processing.

Review Questions

Short Answer Questions

1. What is the principal difference between a command and an event?

2. Why can a synchronous service call create cascading failure?

3. What purpose does an outbox pattern serve?

4. Define an idempotent consumer.

5. Why should a dead-letter queue preserve diagnostic metadata?

Long Answer Questions

1. Compare queues, topics, and event streams, including their appropriate use cases.

2. Explain how timeouts, retries, circuit breakers, and bulkheads improve synchronous communication.

3. Discuss ordering and partitioning in an event-streaming architecture.

4. Compare choreography and orchestration for a long-running business workflow.

5. Explain schema evolution practices that allow producers and consumers to change independently.

Analytical / Application Questions

1. A payment service times out after the provider may have completed the charge. Design a safe retry and idempotency strategy.

2. An organisation wants to publish order updates to fulfilment, analytics, notifications, and fraud detection. Recommend a communication design and justify your choice.

3. A consumer repeatedly fails on one malformed event, blocking later events in its queue. Propose a retry, dead-letter, repair, and replay process that protects both reliability and business correctness.

22. Multiple Choice Questions with Answers

Question 1

Which situation most strongly favours asynchronous communication?

A. A client needs an immediate validation result before displaying a form error.

B. A short read must return current data before a transaction continues.

C. A notification can be processed later and should not delay order acceptance.

D. A caller must receive a definitive response from a single service within a request.

Correct Answer: C. A notification can be processed later and should not delay order acceptance.

Question 2

What is the primary purpose of an outbox pattern?

A. To guarantee global ordering across all services.

B. To atomically persist a local state change and its outgoing event.

C. To eliminate the need for consumer acknowledgements.

D. To convert every event into a synchronous API call.

Correct Answer: B. To atomically persist a local state change and its outgoing event.

Question 3

A consumer receives the same payment-authorisation command twice. Which design best prevents a duplicate charge?

A. Increase the retry count indefinitely.

B. Use a unique idempotency key and persist processing state.

C. Process the second message with a higher priority.

D. Assume the broker provides universal exactly-once execution.

Correct Answer: B. Use a unique idempotency key and persist processing state.

Question 4

What does a partition key such as orderId primarily help achieve in an event stream?

A. Ordering of events related to the same order while allowing parallelism across orders.

B. Global ordering of every event in the system.

C. Automatic schema compatibility between all consumers.

D. Removal of the need for retries.

Correct Answer: A. Ordering of events related to the same order while allowing parallelism across orders.

Question 5

Which characteristic best describes orchestration?

A. Services react to events without any process coordinator.

B. A central process manager directs workflow steps and handles progress.

C. All services write to one shared database table.

D. Consumers ignore failures because events are immutable.

Correct Answer: B. A central process manager directs workflow steps and handles progress.

Question 6

Why is acknowledging a message before completing its business update dangerous?

A. It may cause the broker to reorder unrelated messages.

B. It can cause the message to be lost if the consumer crashes afterward.

C. It prevents all other consumers from reading the topic.

D. It automatically creates duplicate business records.

Correct Answer: B. It can cause the message to be lost if the consumer crashes afterward.

Question 7

Which change is generally most backward-compatible for an event schema?

A. Renaming an existing required field.

B. Changing a monetary field from major units to minor units without notice.

C. Adding an optional field while preserving existing field meanings.

D. Removing a field that older consumers still require.

Correct Answer: C. Adding an optional field while preserving existing field meanings.

Question 8

What is the appropriate role of a dead-letter queue?

A. To store every successfully processed event permanently.

B. To isolate messages that repeatedly fail so they can be investigated and safely handled.

C. To replace all application logging.

D. To guarantee that a malformed message eventually succeeds without intervention.

Correct Answer: B. To isolate messages that repeatedly fail so they can be investigated and safely handled.

Question 9

Which problem is choreography especially likely to create when a workflow becomes very complex?

A. The absence of any possible fan-out.

B. Difficulty understanding distributed business rules and process progression.

C. Inability to publish immutable events.

D. Mandatory global ordering across services.

Correct Answer: B. Difficulty understanding distributed business rules and process progression.

Question 10

A service writes an order update successfully but crashes before publishing its event. Which problem does this illustrate?

A. The dual-write problem.

B. The partition-key problem.

C. The bulkhead problem.

D. The schema-registry problem.

Correct Answer: A. The dual-write problem.