7. Distributed Systems, Microservices and Event-Driven Architecture

Introduction and Context

A backend application begins as a program that responds to requests, stores data, and applies business rules. As its user base, organizational complexity, and operational demands grow, the application often becomes a distributed system: its functionality is divided across processes, machines, databases, networks, and external services. Distribution can improve scalability and organizational autonomy, but it also introduces latency, partial failure, inconsistent data, difficult testing, and more demanding operational responsibilities.

This chapter examines three important architectural directions: the modular monolith, microservices, and serverless systems. It then develops the concepts required to design and operate distributed backends, including bounded contexts, service decomposition, service discovery, API gateways, message brokers, event sourcing, CQRS, sagas, retries, circuit breakers, eventual consistency, and distributed transaction patterns. Examples refer to Node.js, Django, and Spring Boot because the architectural principles apply across these ecosystems even though their implementation mechanisms differ.

The central design question is not whether microservices are modern or whether monoliths are outdated. It is whether the selected architecture creates an appropriate balance among business modularity, delivery speed, reliability, scalability, governance, and operational cost.

Learning Objectives

After completing this chapter, the learner should be able to:

• distinguish a modular monolith, microservices architecture, and serverless architecture;
• use domain boundaries and bounded contexts to identify service candidates;
• evaluate decomposition, service ownership, and database-per-service decisions;
• explain the roles of API gateways, service discovery, and message brokers;
• design workflows using synchronous APIs, asynchronous events, sagas, CQRS, and event sourcing;
• apply retries, timeouts, circuit breakers, idempotency, and dead-letter handling;
• reason about eventual consistency and distributed transaction alternatives;
• assess managerial, security, governance, and operational consequences; and
• select an architecture on the basis of evidence rather than fashion.

From a Monolith to a Distributed Architecture

A monolith is deployed as one unit. This does not necessarily mean that it is poorly designed. A conventional monolith may contain clear modules, stable interfaces, and well-defined ownership. Its main advantage is that calls between modules are local function calls and transactions can often be handled by one database transaction. Deployment, debugging, and testing are comparatively simple.

A modular monolith preserves these advantages while imposing internal boundaries. For example, an e-commerce application may contain Catalog, Ordering, Payments, Inventory, and Shipping modules. Each module can have its own domain model, application services, persistence layer, and tests. Modules communicate through explicit interfaces rather than by directly manipulating one another's tables. This architecture is frequently an effective first step because it allows teams to discover domain boundaries before paying the cost of distribution.

Microservices take the idea of modularity across process boundaries. A service is independently deployable, owns a meaningful business capability, and communicates through APIs or messages. Independent deployment is valuable when different parts of a product have different release frequencies, scaling profiles, risk levels, or team ownership. It is not valuable merely because the application has many classes or because a framework makes HTTP communication easy.

Serverless architecture moves operational responsibility further toward a cloud provider. Functions are invoked by HTTP requests, schedules, queues, or events, while the platform manages much of the server provisioning and scaling. Serverless is particularly suitable for sporadic workloads, event processing, and small independently deployable handlers. However, cold starts, execution limits, provider-specific services, observability challenges, and distributed state remain important concerns. Serverless applications are still distributed systems; the absence of server administration does not remove architectural complexity.

The progression can therefore be understood as a change in deployment and operational boundaries, not as a hierarchy in which each later form is universally superior.

Bounded Contexts and Service Decomposition

Domain-driven design provides a useful vocabulary for decomposition. A bounded context is a boundary within which a particular domain model and vocabulary have consistent meaning. The word “customer,” for example, may refer to a billing account in one context, a delivery recipient in another, and a marketing profile in a third. Treating these meanings as one universal data model often creates coupling and ambiguity.

A service candidate should normally represent a cohesive business capability rather than a technical layer. “Order Management” is a stronger boundary than “Database Service” or “Validation Service.” Useful decomposition signals include:

• a distinct business responsibility and vocabulary;
• data that can be owned by one team or capability;
• a different rate, reliability requirement, or scaling pattern;
• a clear lifecycle and relatively stable interface;
• a need for independent deployment; and
• a manageable set of dependencies.

Strong cohesion and loose coupling are more important than the number of services. Excessive decomposition produces a distributed monolith: services are nominally separate but must be deployed together, make long chains of synchronous calls, share databases, and fail as one system. This arrangement combines the complexity of microservices with few of their benefits.

A practical decomposition process begins with domain events and business workflows. Teams identify activities such as OrderPlaced, PaymentAuthorized, StockReserved, or ShipmentDispatched, then examine which decisions, data, and policies produce or consume each event. The result is a set of candidate contexts that must be validated with domain experts. Technical boundaries should not be treated as final merely because they appear elegant on a diagram.

Database ownership is central. A service that owns a business concept should control writes to the corresponding data. Other services receive information through APIs, events, or replicated read models. Shared databases are sometimes acceptable during migration, but they create hidden coupling: schema changes become coordination events, and one service can bypass another's business rules.

Communication: APIs, Discovery, and Gateways

Synchronous communication, commonly implemented with REST, GraphQL, or gRPC, is appropriate when a caller needs an immediate answer. For example, a checkout service may synchronously ask an inventory service whether a requested item can be reserved. The caller must, however, manage timeout, error, version compatibility, and dependency availability.

Asynchronous communication uses a queue or event broker. The producer sends a message and continues, while a consumer processes it later. This reduces temporal coupling and supports buffering, but introduces eventual consistency, duplicate delivery, ordering questions, and more difficult debugging. A command usually requests an action, whereas an event records that something has happened. “ReserveStock” is a command; “StockReserved” is an event.

Service discovery enables a client or gateway to locate service instances whose addresses may change because of scaling, deployment, or failure. In client-side discovery, the client queries a registry and selects an instance. In server-side discovery, a load balancer or platform performs lookup on the client's behalf. Kubernetes Services, cloud load balancers, and registries such as Consul are examples of mechanisms that can provide this capability.

An API gateway is an edge component that can route requests, terminate TLS, authenticate clients, enforce rate limits, transform protocols, and aggregate responses. It should not become a replacement for domain services or a large repository of business logic. A gateway can also provide a backend-for-frontend arrangement in which mobile and web clients receive representations suited to their needs.

A broker such as Kafka, RabbitMQ, or a cloud queue provides varying combinations of persistence, routing, ordering, replay, delivery guarantees, and consumer coordination. “Exactly once” should be treated carefully: even when a broker provides strong delivery semantics, the surrounding database and external side effects may not be exactly once. Consumers should therefore be idempotent, commonly by recording a message identifier or using a business idempotency key.

Consistency and Distributed Transactions

In a single database transaction, atomicity is relatively straightforward: either all changes commit or none do. Across services, a business operation may involve several databases and external providers. Network failure can occur after one participant commits but before the caller receives a response. Consequently, distributed systems frequently use eventual consistency: replicas or services may temporarily disagree, but a valid process eventually brings them to a consistent state.

Eventual consistency is not the same as careless inconsistency. The system must define acceptable states, convergence mechanisms, user-visible status, and repair procedures. A customer may see “payment verification pending” rather than a false success or an unexplained error. Read models should indicate their freshness when that matters.

Two-phase commit attempts to coordinate a prepare phase and a commit phase among participants. It can provide atomicity but introduces blocking, coordinator dependency, performance overhead, and operational complexity. It is therefore uncommon for internet-scale business workflows crossing heterogeneous services.

The saga pattern divides a long business transaction into local transactions. Each step commits independently. If a later step fails, a compensating action reverses or semantically offsets earlier work. In an orchestration-based saga, a coordinator directs the sequence. In a choreography-based saga, services react to one another's events. Orchestration provides a visible workflow but can centralize control; choreography reduces central coordination but may produce an opaque event network.

Consider an order workflow: create order, reserve stock, authorize payment, and arrange shipment. If payment fails after stock reservation, a compensation can release the stock and mark the order as failed. Compensation is not always a literal undo. A shipment already dispatched cannot be erased; the business may instead create a return or refund process. Sagas therefore require explicit business policies, not merely technical rollback code.

Event Sourcing, CQRS, and Event-Driven Design

Event sourcing stores the sequence of domain events as the authoritative record rather than storing only the latest state. The current state is reconstructed by replaying events, and projections can produce specialized views. For an account, events might include AccountOpened, MoneyDeposited, and MoneyWithdrawn. Event sourcing supports auditability, temporal analysis, and rebuilding projections, but it requires event versioning, storage discipline, replay performance, privacy planning, and careful handling of corrections.

CQRS, or Command Query Responsibility Segregation, separates models used to change state from models used to read state. A command model enforces business rules; one or more query projections are optimized for particular access patterns. CQRS does not require event sourcing, and event sourcing does not automatically require multiple services. The combination is useful where read and write workloads differ significantly or where rich historical events are valuable.

A projection is derived data and may lag behind the source. A robust design records the position of processed events, supports replay, and detects malformed or incompatible events. The outbox pattern helps publish reliable events: a service writes its business change and an outgoing event record in the same local database transaction. A separate publisher then sends the event to the broker and marks it as dispatched. This prevents the failure mode in which the database commits but event publication is lost.

The inbox or idempotent-consumer pattern records consumed message identifiers and ensures that redelivery does not repeat a non-idempotent operation. Together, outbox and inbox patterns address the practical gap between local transactions and message delivery, although they do not eliminate all operational failure modes.

Reliability Patterns for Partial Failure

Every remote call should have a bounded timeout. Without one, threads, event-loop tasks, connection pools, or worker processes can remain occupied while waiting for a failed dependency. Retries are appropriate for transient failures such as temporary network interruption or a throttling response, but not for validation errors or permanent business rejection.

A good retry policy uses a limited number of attempts, exponential backoff, and jitter. Jitter prevents many clients from retrying simultaneously and creating a recovery storm. Retries must be combined with idempotency. Retrying a payment request without a provider idempotency key can charge a customer twice.

A circuit breaker observes failures and moves from closed to open after a threshold. In the open state, calls fail quickly rather than overwhelming an unhealthy dependency. After a recovery interval, a half-open state permits limited test requests. Circuit breakers protect the caller and dependency, but they do not replace root-cause analysis, fallback design, or capacity planning.

Bulkheads isolate resources so that failure in one dependency does not exhaust all worker threads or connections. Rate limiting protects services from overload, while load shedding deliberately rejects low-priority work when capacity is insufficient. Health checks should distinguish liveness from readiness: a live process may be unable to serve traffic because a required dependency is unavailable.

Observability is essential. Logs should include correlation or trace identifiers and structured fields. Metrics should cover latency, error rate, throughput, saturation, queue depth, consumer lag, retry counts, and circuit state. Distributed tracing follows a request across services and helps identify whether delay arises in the gateway, application, database, or broker. Observability data must itself be governed because traces and logs may contain personal or confidential information.

Applying the Architecture in Node.js, Django, and Spring Boot

In Node.js, asynchronous I/O and worker-based consumers make it natural to build API services and event processors. Frameworks such as Express or NestJS can expose HTTP endpoints, while broker clients handle Kafka, RabbitMQ, or cloud queues. Developers must still control promise failures, connection pooling, backpressure, and CPU-intensive work; asynchronous syntax does not make a service automatically resilient.

Django provides routing, authentication, ORM support, and administrative facilities for service development. Django applications can publish domain events through an outbox table and process asynchronous work with systems such as Celery and a message broker. Care is required when ORM transactions and task publication interact: a task should not be published before the associated database transaction has committed.

Spring Boot offers dependency injection, web support, transaction management, Actuator metrics, and integration with messaging systems. Spring Cloud components can assist with configuration, discovery, gateway functions, and resilience patterns, although platform-native mechanisms may be preferable in some deployments. In all three ecosystems, framework features support but do not determine domain boundaries, ownership, idempotency, or compensation policy.

Illustrative Case Study: An Online Retail Platform

Background

An illustrative retailer began with a Django monolith serving a web storefront. The system used one relational database for catalog, orders, customers, inventory, and payment records. The application was reliable at moderate scale, but releases became risky because a change to inventory could affect checkout and reporting. Product search also generated heavy database load during marketing campaigns.

Situation and Analysis

The organization initially proposed splitting every major database table into a service. Analysis showed that this would create excessive synchronous calls and preserve the old coupling through shared tables. Domain workshops instead identified Catalog, Order Management, Inventory, Payment Integration, and Shipping as bounded contexts. Reporting was treated as a read concern rather than as an independent owner of transactional data.

The team selected a modular monolith as an intermediate state. Each context received explicit interfaces and tests, and direct cross-module database access was prohibited. Catalog search was separated first because its read-heavy workload and indexing technology differed from transactional ordering. Later, Order Management and Inventory became independently deployable services.

Approach and Solution

The customer request entered through an API gateway. The gateway authenticated the client and routed the request to Order Management. The order service created an order with status “pending” and wrote an OrderCreated event to an outbox table in the same transaction. A publisher delivered the event to the broker.

Inventory consumed the event and attempted a local reservation. It emitted StockReserved or StockReservationFailed. Payment Integration then authorized payment only after stock reservation. The saga coordinator updated order status as events arrived. If payment failed, it issued a compensating ReleaseStock command. The user interface displayed explicit states such as pending, confirmed, and failed rather than pretending that all operations were instantaneous.

Consumers stored event identifiers to achieve idempotency. HTTP calls used deadlines, bounded retries, and provider idempotency keys. A circuit breaker prevented a payment provider outage from exhausting order-processing resources. A query projection combined order status, item details, and shipment information for fast customer views; the projection was acknowledged as eventually consistent.

Outcome

The architecture reduced the deployment scope of inventory changes and isolated search scaling from transactional checkout. It also introduced broker monitoring, replay procedures, reconciliation jobs, schema compatibility rules, and operational training. The case demonstrates that the principal gain came from appropriate ownership and controlled coupling, not from the number of services.

Learning Points

First, a modular monolith can reveal boundaries before distribution. Second, a saga expresses business recovery rather than providing magical rollback. Third, reliable events require local transactional recording, idempotent consumers, and operational observability. Finally, user experience must communicate intermediate states honestly.

Risks, Governance, and Professional Responsibilities

Distributed systems increase the attack surface. Each service needs authentication and authorization, secure secret handling, transport encryption, input validation, dependency patching, and least-privilege access to data and brokers. An API gateway must not be the only security layer; internal service-to-service authorization is necessary when trust boundaries demand it.

Data duplication creates privacy and retention obligations. Event logs and projections may contain personal information that must be deleted, masked, or access-controlled under applicable law and organizational policy. Event immutability can conflict with erasure requirements, so teams should avoid placing unnecessary personal data in immutable events and should design anonymization or cryptographic-reference strategies.

Governance should cover API and event versioning, ownership, service-level objectives, incident response, change approval proportional to risk, cost allocation, and decommissioning. Teams need a service catalogue, dependency map, runbooks, and recovery procedures. Architectural autonomy without common standards can produce incompatible protocols and duplicated infrastructure; excessive central control can destroy the independence that services were intended to provide.

Common mistakes include creating services around technical layers, sharing databases, using synchronous chains for every operation, retrying non-idempotent commands, assuming exactly-once processing, ignoring schema evolution, and treating asynchronous messaging as a solution to every performance problem. A service should be extracted only when its boundary, operational rationale, and ownership are sufficiently understood.

Current Directions and Best Practices

Platform engineering increasingly packages common capabilities such as deployment, secrets, telemetry, policy enforcement, and service templates. Kubernetes and managed cloud services can reduce operational effort, but they also introduce platform complexity and costs. Managed brokers, serverless functions, and workflow engines are useful when their operational model matches the workload.

Modern systems also apply stream processing and real-time analytics to events, while machine-learning components consume event streams for recommendation, fraud detection, and anomaly analysis. Such uses require careful attention to data quality, model drift, explainability, and the distinction between an analytical event stream and an authoritative business transaction.

Effective practice can be summarized as follows:

1. begin with business capabilities and bounded contexts;

2. prefer a well-structured modular monolith when independent deployment is not yet justified;

3. assign clear data ownership and avoid shared writes;

4. choose synchronous or asynchronous communication according to business timing and failure needs;

5. make consumers idempotent and use outbox or equivalent reliability patterns;

6. define timeouts, retry budgets, circuit breakers, and compensation policies;

7. design observability and operational recovery before production deployment; and

8. measure architecture through delivery performance, reliability, customer outcomes, and total cost.

Chapter Summary

Distributed architecture is a set of trade-offs rather than a mandatory destination. A modular monolith can provide strong internal boundaries with relatively simple operations. Microservices enable independent deployment and scaling when bounded contexts, ownership, and organizational capacity justify them. Serverless changes infrastructure responsibility but retains the problems of distributed state, failure, security, and observability.

Bounded contexts guide service decomposition; APIs, gateways, discovery mechanisms, and brokers support communication; and database ownership prevents hidden coupling. Because distributed transactions are difficult, systems commonly use eventual consistency, sagas, compensating actions, outbox and inbox patterns, CQRS, and, where appropriate, event sourcing. Timeouts, bounded retries, jitter, circuit breakers, bulkheads, idempotency, and tracing make partial failure manageable. Sound architecture combines technical patterns with governance, privacy, security, clear ownership, and honest user-facing states.

Key Terms

Bounded context: A domain boundary within which a model and vocabulary have consistent meaning.

Modular monolith: A single deployable application organized into strongly isolated business modules.

Microservice: An independently deployable service centered on a cohesive business capability.

Serverless: An execution model in which the platform manages much of the server provisioning and scaling.

Service discovery: The mechanism by which clients or infrastructure locate available service instances.

API gateway: An edge component that routes, secures, limits, or aggregates client requests.

Message broker: Infrastructure that stores, routes, and delivers messages between producers and consumers.

Command: A message requesting that a recipient perform an action.

Event: A record that a business fact has occurred.

Eventual consistency: A model in which distributed representations may temporarily differ but converge over time.

Saga: A sequence of local transactions coordinated through forward actions and compensations.

Event sourcing: Persisting domain events as the authoritative record of state changes.

CQRS: Separating models responsible for changing state from models optimized for reading state.

Projection: A derived read model built from events or other authoritative data.

Outbox pattern: Recording a business change and outgoing event in one local transaction before publication.

Idempotency: The property that repeating an operation produces the same intended result as performing it once.

Circuit breaker: A control that stops calls to an unhealthy dependency for a period of time.

Bulkhead: Resource isolation that limits the impact of failure in one dependency or workload.

Distributed tracing: Tracking a request across multiple processes using correlated trace information.

Consumer lag: The delay between event production and its processing by a consumer.

Review Questions

Short Answer Questions

1. What distinguishes a modular monolith from a microservices architecture?

2. Why is bounded context more useful than a database table for identifying service boundaries?

3. Differentiate a command from an event.

4. What problem does the outbox pattern address?

5. Why must retries normally be combined with idempotency?

Long Answer Questions

1. Explain the criteria and process for decomposing a modular monolith into microservices.

2. Discuss eventual consistency, sagas, and compensating actions in a distributed order workflow.

3. Compare event sourcing and CQRS, including situations in which each may be appropriate.

4. Explain how timeouts, retries, circuit breakers, bulkheads, and observability work together to manage partial failure.

5. Evaluate the organizational, security, privacy, and governance consequences of adopting microservices.

Analytical / Application Questions

1. A payment service is unavailable after inventory has been reserved. Design a saga response, including user-visible states, compensation, retry policy, and idempotency controls.

2. An organization has six teams but one shared database and tightly coupled releases. Recommend whether it should adopt microservices immediately, remain a modular monolith, or use an intermediate strategy. Justify the decision.

3. A high-volume event consumer processes messages twice and occasionally produces duplicate shipments. Diagnose the likely design weaknesses and propose an outbox, inbox, idempotency, and reconciliation strategy.

22. Multiple Choice Questions with Answers

Question 1

Which situation most strongly justifies extracting a module from a modular monolith into an independent service?

A. The module contains many classes

B. The module has a distinct business capability, ownership, and scaling or deployment requirement

C. The module uses a different programming language for convenience

D. The database table owned by the module has many columns

Correct Answer: B. The module has a distinct business capability, ownership, and scaling or deployment requirement

Question 2

What is the principal purpose of a bounded context?

A. To guarantee that every service uses the same domain model

B. To define a boundary within which a domain vocabulary and model have consistent meaning

C. To replace authentication between services

D. To ensure that all services share one database

Correct Answer: B. To define a boundary within which a domain vocabulary and model have consistent meaning

Question 3

An order service writes an order and an outgoing event in the same local database transaction, then publishes the event later. Which pattern is being used?

A. Two-phase commit

B. Circuit breaker

C. Outbox pattern

D. Client-side discovery

Correct Answer: C. Outbox pattern

Question 4

Why is a circuit breaker useful when a downstream service is failing?

A. It guarantees that the downstream service will recover

B. It prevents repeated calls from consuming excessive caller resources

C. It converts all synchronous calls into events

D. It makes database transactions globally atomic

Correct Answer: B. It prevents repeated calls from consuming excessive caller resources

Question 5

Which statement best describes a saga?

A. A single database transaction spanning every service

B. A sequence of local transactions with compensating actions for failure

C. A cache that stores service-discovery results

D. A protocol that guarantees exactly-once external side effects

Correct Answer: B. A sequence of local transactions with compensating actions for failure

Question 6

A payment request may be retried after a network timeout. Which control most directly reduces the risk of charging the customer twice?

A. A larger connection pool

B. A provider-supported idempotency key

C. A longer request timeout

D. A shared database between payment and order services

Correct Answer: B. A provider-supported idempotency key

Question 7

What is a defining characteristic of CQRS?

A. All services must use event sourcing

B. Read and write responsibilities use separated models or paths

C. Every command must be processed synchronously

D. Queries are prohibited from using projections

Correct Answer: B. Read and write responsibilities use separated models or paths

Question 8

Which design best reflects eventual consistency?

A. A customer sees an explicit pending status while a payment result is being processed

B. All services lock one database before accepting any request

C. A gateway hides every failure and always returns success

D. A service directly edits another service's tables

Correct Answer: A. A customer sees an explicit pending status while a payment result is being processed

Question 9

What is the main danger of a long chain of synchronous service calls?

A. It eliminates the need for monitoring

B. It increases temporal coupling and allows one failure to propagate through the chain

C. It prevents all independent deployment

D. It guarantees stale read models

Correct Answer: B. It increases temporal coupling and allows one failure to propagate through the chain

Question 10

Which statement about event sourcing is accurate?

A. It stores only the latest state and discards history

B. It treats domain events as the authoritative record from which state can be reconstructed

C. It automatically provides distributed atomic transactions

D. It removes the need for event versioning

Correct Answer: B. It treats domain events as the authoritative record from which state can be reconstructed