9. Integration and Event-Driven Architecture

Introduction and Context

Modern applications rarely operate as isolated programs. A banking platform may depend on identity verification, payment gateways, fraud detection, notification services, and regulatory reporting. An online marketplace connects customers, sellers, inventory systems, logistics providers, and payment processors. In each case, the quality of the overall system depends not only on the internal design of individual services but also on how those services communicate and coordinate.

Integration architecture provides the structures and rules through which software components exchange data and trigger actions. Two broad styles dominate contemporary full-stack systems. In synchronous integration, a caller sends a request and waits for a response. In asynchronous integration, a producer sends a message or event and continues without waiting for the consumer to complete its work. Neither style is universally superior. Synchronous communication is often simple and appropriate for immediate validation, while asynchronous communication improves decoupling, resilience, and scalability when immediate responses are unnecessary.

Event-driven architecture extends asynchronous integration by treating events—records of facts that have occurred—as important architectural elements. This approach introduces powerful capabilities, but also creates difficult issues involving ordering, duplicate delivery, data consistency, observability, security, and failure recovery. At postgraduate level, the central question is not whether an organisation should use events, but where event-driven techniques create sufficient value to justify their operational and conceptual complexity.

Learning Objectives

By the end of this chapter, the learner should be able to:

• distinguish synchronous requests, commands, messages, and events;
• explain queues, publish–subscribe systems, event streaming, event sourcing, and CQRS;
• select suitable integration styles according to latency, coupling, consistency, and failure requirements;
• apply Saga, Outbox, Circuit Breaker, timeout, retry, and idempotency patterns;
• design robust message contracts and address ordering, duplication, replay, and schema evolution;
• analyse eventual consistency and integration failure in distributed systems;
• evaluate professional, security, governance, and operational implications of event-driven architecture.

The Integration Vocabulary: Requests, Commands, Messages, and Events

A request asks another component to perform an operation and normally expects a response. In a REST interaction, for example, an order service may request the current credit limit from an account service. Requests are commonly synchronous, although they may also be transported asynchronously.

A command expresses an intention: “ReserveStock,” “IssueRefund,” or “CreateShipment.” It is directed to a particular handler, which may accept or reject it. A command should not be confused with an event. An event states that something has already happened, such as “StockReserved” or “RefundIssued.” Events are facts rather than instructions and may be consumed by many independent subscribers.

A message is the transportable representation of information exchanged between components. Commands and events are both kinds of messages. A message generally includes a payload and metadata such as an identifier, correlation identifier, causation identifier, timestamp, producer, schema version, tenant, and trace context.

This distinction has architectural consequences. A command creates a dependency on a specific capability or receiver. An event creates a subscription relationship and allows several consumers to react independently. Poorly naming commands as events can conceal control coupling; conversely, treating every notification as a command can produce unnecessary central orchestration.

Choosing Between Synchronous and Asynchronous Integration

Synchronous integration is appropriate when the caller cannot proceed without an immediate answer. Examples include authenticating a user, validating a form, retrieving a product page, or checking whether a payment request is accepted. Its advantages include straightforward control flow, immediate error reporting, and relatively simple debugging. Its weaknesses are temporal coupling and failure propagation: if the downstream service is slow or unavailable, the caller may also become slow or unavailable.

Asynchronous integration separates the time of sending from the time of processing. A producer places a message on a queue or publishes an event, and a consumer processes it later. This supports workload smoothing, independent scaling, and resilience to temporary downstream unavailability. It is suitable for sending emails, generating reports, updating search indexes, processing images, and propagating business facts.

The trade-off is that the producer usually cannot provide an immediate final result. Consumers may fail, messages may be delivered more than once, and different views of the business may temporarily disagree. Asynchronous systems therefore require explicit operational mechanisms for retries, dead-letter handling, monitoring, reconciliation, and idempotent processing.

A useful decision framework considers five questions:

1. Is an immediate business answer required?

2. What is the acceptable latency and consistency model?

3. Can the work be safely repeated?

4. What should happen if the receiving service is unavailable?

5. Does the benefit of decoupling justify more complex operations?

Queues, Publish–Subscribe, and Event Streams

A queue normally distributes work among competing consumers. If five workers consume an image-processing queue, each message is generally processed by one worker. Queues are useful for load balancing, background jobs, and controlling the rate at which a fragile dependency is called. Acknowledgement mechanisms determine when a message is considered successfully processed. If a consumer fails before acknowledgement, the message may be redelivered.

A publish–subscribe system distributes a published event to multiple subscribers. The billing service, notification service, analytics service, and audit service may all receive an “OrderCompleted” event independently. This reduces direct coupling between the order service and each consumer. Subscribers must, however, manage their own processing state and failures.

Event-streaming platforms retain ordered records for a period or indefinitely. Consumers track offsets and can replay records to rebuild a projection or recover from an error. Partitioning permits parallel processing, but ordering is generally guaranteed only within a partition. A design that requires all events for one customer or order to be processed sequentially should use a stable partitioning key such as customer ID or order ID.

Delivery semantics must be stated precisely. At-most-once delivery may lose messages but avoids redelivery. At-least-once delivery avoids silent loss more effectively but permits duplicates. Exactly-once claims are often limited to a particular platform boundary and processing model; they should not be assumed to extend automatically across databases and external services. In practice, durable messaging combined with idempotent consumers is a robust general strategy.

Event-Driven Architecture as a Design Approach

In an event-driven architecture, important changes in state are represented as events that other components can observe. Producers do not need to know all current consumers. This promotes autonomy and permits new capabilities—such as fraud analysis or customer analytics—to subscribe without changing the original producer.

Event-driven architecture has two related but distinct forms. In event notification, an event contains limited information and consumers retrieve further details if required. In event-carried state transfer, the event contains enough information for consumers to update their local representation without an additional request. The latter reduces synchronous dependencies but increases concerns about payload size, privacy, and stale information.

A well-designed event should have a stable business meaning, a clear owner, an explicit schema, and documented compatibility rules. Events should describe domain facts rather than database implementation details. For example, “OrderPaid” is usually more durable and meaningful than “OrdersTableRowUpdated.”

Event Sourcing and CQRS

Traditional persistence stores the current state of an entity. Event sourcing instead stores the sequence of domain events that led to that state. The current state is reconstructed by replaying events, while snapshots may be taken periodically to reduce reconstruction cost. Event sourcing can provide a detailed history, support temporal analysis, and enable new projections to be built from past facts.

It also introduces significant obligations. Event schemas become long-lived contracts. A correction cannot simply erase history; it generally requires a compensating event. Replay performance, privacy deletion requirements, storage growth, and migration of old event versions must be designed from the beginning. Event sourcing is therefore most valuable where business history is itself important, such as financial movements, claims, or workflow decisions—not merely because an organisation uses a message broker.

CQRS, or Command Query Responsibility Segregation, separates the model that handles state-changing commands from the model that serves queries. The write model enforces business invariants, while one or more read projections are optimised for particular user interfaces or reporting needs. CQRS can improve query performance and allow different storage technologies, but it introduces projection lag and operational complexity.

Event sourcing and CQRS are independent choices. An application may use CQRS with ordinary transactional persistence, or event sourcing without multiple query models. Combining them is powerful but should be justified by domain complexity, audit needs, and read-model requirements.

Reliable Integration Patterns

The Outbox Pattern

A central failure problem occurs when a service updates its database and publishes an event as two separate operations. The database transaction may commit while event publication fails, or the event may be published while the transaction rolls back. The Outbox pattern solves this by writing the business change and an outgoing message to an outbox table in the same local transaction. A relay process later publishes the outbox record to the broker and marks it as dispatched.

The relay may publish the same record more than once, so consumers must be idempotent. Outbox records also require retention, monitoring, cleanup, and a policy for permanently failed messages. The pattern provides atomicity between local state and the intent to publish; it does not create a global transaction across all services.

Saga and Compensation

A distributed business transaction often spans several services. A travel booking may reserve a seat, charge a card, and arrange accommodation. A two-phase commit protocol can coordinate resources but may reduce availability and increase operational coupling. A Saga divides the process into local transactions. Each successful step is followed by the next action; if a later step fails, compensating actions reverse or neutralise earlier effects.

A Saga may be orchestrated, with a coordinator directing participants, or choreographed, with participants reacting to events. Orchestration makes the process visible and easier to govern, but can create a central coordinator. Choreography reduces central control but may become difficult to understand as the number of participants grows. Compensation is not always a literal rollback: a refund, cancellation, or corrective entry may be the appropriate business response.

Circuit Breaker, Timeouts, and Retries

A Circuit Breaker prevents repeated calls to a failing dependency. In the closed state, calls proceed normally. After failures exceed a threshold, the circuit opens and calls fail quickly or use a fallback. After a cooling period, a half-open state allows limited test calls. This protects both caller and receiver from cascading failure.

Timeouts are essential because an absent timeout can consume threads, connections, and memory indefinitely. Retries are useful for transient failures, but indiscriminate retries amplify load and may duplicate non-idempotent actions. Good retry policies use bounded attempts, exponential backoff, and jitter. They distinguish transient errors, such as temporary network loss, from permanent errors, such as invalid data. A retry should never replace proper business-level recovery.

Idempotency and Deduplication

An operation is idempotent if applying it repeatedly has the same intended business effect as applying it once. Consumers can achieve this with an idempotency key, a processed-message table, a unique business constraint, or a state transition that ignores already-completed work. For example, processing “PaymentCaptured” should not charge a card again merely because the event was delivered twice.

Deduplication records must be stored durably and evaluated within an appropriate transaction boundary. In some cases, the business operation and the deduplication marker can be committed together. External side effects require additional care; an idempotency key accepted by the payment provider may be necessary.

Contracts, Ordering, and Eventual Consistency

Integration contracts define the structure and meaning of communication. A contract includes field names, types, required and optional values, semantic rules, error behaviour, versioning policy, and ownership. Contract tests can verify that a producer remains compatible with consumer expectations without requiring a complete end-to-end environment.

Backward-compatible evolution usually adds optional fields rather than renaming or removing existing ones. Consumers should tolerate unknown fields and producers should avoid changing the meaning of an existing field. When incompatible change is unavoidable, a new event version or parallel contract may be required. Schema registries, validation pipelines, and consumer-driven contract testing support disciplined evolution.

Ordering is meaningful only when the business process requires it. A broker may preserve order within a partition but not across partitions or unrelated consumers. Events should carry sequence numbers or versions when consumers need to detect gaps or stale updates. A consumer that receives version 8 after version 10 may defer processing, request reconciliation, or rebuild its projection.

Eventual consistency means that a successful change in one component becomes visible in other components after a delay. It is not the same as inconsistency without control. The system should define acceptable lag, display states such as “processing,” reconciliation procedures, and ownership of authoritative data. User interfaces should not imply finality when an operation is still being completed asynchronously.

Applied Case: An Illustrative Order Fulfilment Platform

Background

Consider an illustrative online retailer composed of Order, Inventory, Payment, Shipping, Notification, and Analytics services. The retailer wants customers to receive rapid order confirmation while preventing overselling and avoiding a single central transaction across all services.

Situation and Problem

A first design synchronously calls inventory, payment, and shipping from the Order service. During promotional traffic, inventory becomes slow. Order requests remain open, threads are exhausted, and retries occasionally cause duplicate payment attempts. Furthermore, an order marked “confirmed” sometimes has no corresponding shipping request because the final call fails.

Analysis

The customer needs an immediate acknowledgement that the order has been accepted, but shipping and notification do not need to complete before that acknowledgement. Inventory reservation and payment authorisation are business steps that may succeed or fail independently. The system therefore needs a clear lifecycle rather than a single ambiguous Boolean status.

Approach

The Order service stores the order and an outbox record in one local transaction, initially assigning the status `PENDING`. An `OrderPlaced` event is published. A Saga coordinator requests inventory reservation and payment authorisation. Each service uses an idempotency key derived from the order and operation. Successful actions produce `InventoryReserved` and `PaymentAuthorised`; failures produce explicit rejection events.

If both required steps succeed, the coordinator records `OrderConfirmed` and requests shipment creation. If payment succeeds but inventory cannot be reserved, the Saga issues a compensating refund or void operation. If a downstream service is unavailable, bounded retries with backoff are used, after which the task enters a retry queue or dead-letter queue for operational review. Circuit breakers prevent an unhealthy payment provider from overwhelming the order service.

The customer interface displays “Order received—confirmation in progress” until the business outcome is known. A read model subscribes to order events and supports customer tracking. Analytics consumes events independently, while audit records preserve the lifecycle and correlation identifiers.

Outcome

The design removes long synchronous chains, makes intermediate states explicit, and prevents duplicate processing through idempotency. It does not eliminate failure; instead, it makes failure observable and recoverable. Operations staff can inspect stuck Sagas, replay safe events, and reconcile the read model with the authoritative order state.

Learning Points

The case demonstrates that asynchronous architecture is not simply the insertion of a queue. It requires a business process model, compensation rules, stable contracts, observability, and a user experience that reflects eventual consistency. It also shows why an Outbox pattern and idempotent consumers are complementary rather than optional refinements.

Operational, Security, and Governance Considerations

Event-driven systems require distributed tracing. Correlation and causation identifiers should follow work across HTTP requests, brokers, consumers, retries, and compensations. Useful metrics include consumer lag, processing latency, retry counts, dead-letter volume, event age, failure rate, circuit state, and Saga completion time. Logs should identify message and business keys without exposing sensitive payloads.

Messages may contain personal, financial, or confidential information. Access control must apply to topics, queues, producers, and consumers. Encryption in transit and at rest, key management, tenant isolation, retention limits, and audit trails are essential. Event immutability can conflict with privacy obligations: a retained event may contain personal data that later must be deleted or anonymised. Data minimisation and tokenisation should therefore be considered before events are published.

Governance should define event ownership, naming conventions, schema lifecycle, retention, replay authority, compatibility rules, and incident responsibilities. Uncontrolled event publication creates an event mesh that is difficult to understand and may expose internal implementation details. Architecture review should ask whether an event represents a stable business fact, who is authorised to consume it, and what happens when the producer changes.

Common Failure Modes and Design Errors

A frequent error is using asynchronous messaging to conceal a poorly defined process. Queues do not resolve unclear ownership or contradictory business rules. Another is assuming that message delivery implies business completion. A published command only indicates that an intention was transmitted; the consumer may still reject it.

Designers also commonly omit timeouts, use unlimited retries, ignore poison messages, or place non-idempotent operations behind at-least-once delivery. Treating event order as globally guaranteed is dangerous, as is making every service maintain a copy of every data field. Event sourcing is sometimes adopted without a retention and migration strategy, creating expensive historical data problems.

Testing must include duplicate delivery, delayed messages, out-of-order events, consumer crashes, broker unavailability, schema changes, partial Saga completion, and replay. Resilience testing and controlled fault injection are particularly valuable because ordinary functional tests rarely expose distributed timing failures.

Current Directions in Integration Architecture

Cloud-native systems increasingly combine synchronous APIs, managed queues, event brokers, workflow engines, and stream-processing platforms. The emerging pattern is not the replacement of one style by another, but deliberate hybrid integration. Workflow orchestration can make long-running Sagas visible, while stream processing supports near-real-time analytics and fraud detection.

Serverless consumers can scale in response to event volume, although cold starts, concurrency limits, cost variability, and debugging complexity must be evaluated. Schema registries and automated contract checks are becoming standard components of mature integration platforms. AI-assisted operations may help classify incidents, detect unusual lag, or suggest correlations across traces, but automated decisions require human oversight, explainability, and controls against misleading data.

Best-Practice Guidance

A sound integration design should begin with business boundaries and failure consequences rather than with a particular broker. Define the authoritative owner of each fact, select synchronous communication only where immediate results matter, and make asynchronous workflows explicit. Use durable contracts, correlation identifiers, timeouts, bounded retries, circuit breakers, dead-letter handling, and idempotent processing. Adopt the Outbox pattern where local database changes must reliably produce messages. Use Sagas when a business process spans independent transactional boundaries, and document compensation as a business decision. Finally, operate the system as a distributed system: measure lag, trace causation, rehearse failure, protect sensitive data, and provide reconciliation mechanisms.

Chapter Summary

Integration architecture connects independent application components through synchronous calls, asynchronous messages, queues, subscriptions, and event streams. Synchronous communication offers immediate responses but creates temporal coupling; asynchronous communication improves decoupling and resilience while introducing eventual consistency and operational complexity. Events represent facts, whereas commands express intentions.

Event sourcing stores business history as events, and CQRS separates write processing from query models; both are specialised choices rather than mandatory features of event-driven systems. The Outbox pattern improves reliable publication, Sagas coordinate distributed business processes through local transactions and compensation, and Circuit Breakers, timeouts, retries, and idempotency limit the effects of failure. Durable contracts, schema evolution, ordering strategies, deduplication, tracing, security, governance, and reconciliation are essential to professional implementation.

Key Terms

Synchronous integration: Communication in which the caller waits for a response.

Asynchronous integration: Communication in which sending and processing occur independently.

Message: A transportable unit of information exchanged between components.

Command: An instruction requesting a specific action.

Event: A record of a fact that has already occurred.

Queue: A messaging structure that distributes work among competing consumers.

Publish–subscribe: A model in which multiple subscribers receive a published message.

Event stream: An ordered, durable sequence of records that consumers can process or replay.

Event sourcing: Persisting domain changes as an append-only sequence of events.

CQRS: Separating command processing from query-serving models.

Eventual consistency: A model in which replicated views converge after a delay.

Outbox pattern: Storing a business change and outgoing message in one local transaction.

Saga: A sequence of local transactions coordinated through forward actions and compensation.

Compensating action: A business operation that offsets or neutralises an earlier action.

Circuit Breaker: A mechanism that stops calls to a failing dependency temporarily.

Idempotency: The property that repeated application has the same intended effect as one application.

Dead-letter queue: A destination for messages that cannot be successfully processed.

Consumer lag: The delay between message production and consumer processing.

Contract testing: Testing compatibility between message producers and consumers.

Correlation identifier: A value used to connect related requests, events, logs, and traces.

Review Questions

Short Answer Questions

1. Distinguish a command from an event.

2. Why can at-least-once delivery produce duplicate business effects?

3. What problem does the Outbox pattern address?

4. Define eventual consistency.

5. What are the principal states of a Circuit Breaker?

Long Answer Questions

1. Compare synchronous and asynchronous integration in relation to latency, coupling, failure, and user experience.

2. Explain event sourcing and CQRS, including circumstances in which each is justified.

3. Describe how a Saga coordinates a distributed business process and why compensation is not always a literal rollback.

4. Discuss contract evolution, ordering, replay, and duplicate delivery in an event-driven system.

5. Explain the operational and governance requirements of a production event-driven architecture.

Analytical / Application Questions

1. Design an integration approach for a university examination-result system in which marks are entered by several departments, results must be audited, and students should receive notifications without delaying mark entry.

2. A payment consumer receives the same `PaymentAuthorised` event three times, while a shipping consumer receives events out of order. Propose controls for both problems.

3. An organisation wants to replace all REST calls with events to improve scalability. Evaluate this proposal and recommend a hybrid architecture with justification.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an event from a command?

A. An event requests a future action, while a command records completed work.

B. An event records a fact that has occurred, while a command expresses an intention.

C. An event can have only one consumer, while a command must have many consumers.

D. An event is always synchronous, while a command is always asynchronous.

Correct Answer: B. An event records a fact that has occurred, while a command expresses an intention.

Question 2

A service updates its database successfully but crashes before publishing the corresponding event. Which pattern most directly addresses this problem?

A. Circuit Breaker

B. CQRS

C. Outbox

D. Retry with jitter

Correct Answer: C. Outbox

Question 3

Why must an at-least-once message consumer usually be idempotent?

A. Because the broker guarantees global ordering.

B. Because the same message may be delivered more than once.

C. Because consumers cannot acknowledge messages.

D. Because at-least-once delivery prevents failures.

Correct Answer: B. Because the same message may be delivered more than once.

Question 4

In a Saga, what is the purpose of a compensating action?

A. To increase the number of messages processed per second.

B. To change an immutable event into a command.

C. To offset or neutralise the effect of an earlier successful step.

D. To guarantee a global database transaction.

Correct Answer: C. To offset or neutralise the effect of an earlier successful step.

Question 5

What is the principal effect of an open Circuit Breaker?

A. It sends every failed request repeatedly until success.

B. It prevents calls to a failing dependency for a period.

C. It guarantees exactly-once delivery.

D. It reorders all messages by timestamp.

Correct Answer: B. It prevents calls to a failing dependency for a period.

Question 6

A stream-processing application requires events for the same order to remain sequential while different orders are processed in parallel. Which design is most appropriate?

A. Use the order ID as a partitioning key.

B. Publish every event to a separate global queue.

C. Remove timestamps from all events.

D. Use random partitioning for maximum distribution.

Correct Answer: A. Use the order ID as a partitioning key.

Question 7

Which situation is the strongest justification for event sourcing?

A. A simple read-only product catalogue with no audit requirement.

B. A domain where the historical sequence of business decisions must be preserved and replayed.

C. A system that requires all services to share one database transaction.

D. A user interface that needs only the current value of a field.

Correct Answer: B. A domain where the historical sequence of business decisions must be preserved and replayed.

Question 8

A retry policy for a temporary downstream failure should generally include which combination?

A. Unlimited immediate retries and no timeout.

B. Bounded attempts, exponential backoff, and jitter.

C. Random data mutation and permanent suppression.

D. A retry only after deleting the original message.

Correct Answer: B. Bounded attempts, exponential backoff, and jitter.

Question 9

What does eventual consistency require the system to make explicit?

A. That every read is globally current at all times.

B. Acceptable propagation delay, intermediate states, and reconciliation procedures.

C. That all services use the same database schema.

D. That asynchronous communication is prohibited.

Correct Answer: B. Acceptable propagation delay, intermediate states, and reconciliation procedures.

Question 10

Which practice best supports safe evolution of an event contract?

A. Changing the meaning of an existing field without notifying consumers.

B. Removing all old fields immediately after deployment.

C. Adding compatible optional fields and validating producer–consumer compatibility.

D. Embedding database table names as the permanent event contract.

Correct Answer: C. Adding compatible optional fields and validating producer–consumer compatibility.