Distributed Data Management

Introduction

Microservices architecture distributes business capabilities across independently deployable services. This distribution improves team autonomy and scalability, but it changes the nature of data management. In a monolithic application, a single database and one transaction manager can often enforce consistency across the whole system. In a microservices system, data is usually distributed across services, databases, regions, and sometimes different storage technologies. A business operation may therefore require coordination without relying on one global transaction.

Distributed data management is the discipline of designing ownership, storage, consistency, integration, recovery, and query strategies for data distributed among services. The central design question is not simply “Which database should be used?” It is: “Which service owns each business fact, and how can other services use that fact safely?”

This chapter examines database-per-service, polyglot persistence, transaction boundaries, sagas, compensating actions, transactional outbox and change-data-capture patterns, CQRS, event sourcing, materialized views, caching, and data governance. The emphasis is on selecting appropriate patterns rather than treating any pattern as universally superior.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain data ownership and the database-per-service principle.
2. Distinguish local transactions from distributed business transactions.
3. Design choreography-based and orchestration-based sagas.
4. Evaluate compensating actions, idempotency, retries, and failure handling.
5. Explain the transactional outbox and change-data-capture patterns.
6. Compare CQRS, event sourcing, and materialized views.
7. Select caching strategies while managing staleness and invalidation.
8. Analyze a distributed-data design for consistency, operational, security, and regulatory risks.

1. Data Ownership and Database-per-Service

A service should own the data required to implement its business capability. Ownership means that the service defines the data model, validates changes, controls writes, and publishes information for legitimate consumers. Other services should not directly modify its tables or depend on its internal schema.

The database-per-service principle can be implemented physically in several ways:

• A separate database server or cluster for each service.
• Separate schemas or credentials within one database platform.
• Separate logical stores, such as a relational database for orders and a document store for catalog data.

Physical separation provides stronger isolation, but logical separation can be appropriate during early development or when operational constraints make many database servers impractical. The essential rule is ownership, not the number of machines.

Direct cross-service database access creates hidden coupling. A schema migration in one service may break another service, and a local query may bypass validation or authorization. Instead, a service should expose an API, domain event, or explicitly governed data-sharing interface.

Data ownership does not mean that information can never be copied. A shipping service may store a read-only delivery projection containing an address supplied by an order service. The order service remains the authority for the order’s address at the relevant business point in time. The copy is a consumer-owned representation, not shared ownership.

Bounded contexts and authoritative facts

Domain-driven design describes a bounded context as a boundary within which terms and models have consistent meanings. “Customer,” “account,” and “product” may legitimately have different attributes in different contexts. A customer service may own identity data; a billing service may own credit status; a marketing service may own preferences.

A useful ownership test is:

1. Which service decides whether a change is valid?
2. Which service is accountable for the business rule?
3. Which service records the authoritative history?
4. Which service is responsible for correcting errors?

The answers identify the system of record. Other services may maintain projections, caches, or snapshots, but should not silently become competing authorities.

2. Polyglot Persistence

Polyglot persistence uses different storage technologies for different requirements. A relational database may support strong constraints and complex joins for payments. A document database may suit independently evolving product descriptions. A key-value store may provide low-latency sessions, while a search engine may support text retrieval.

Technology choice should follow workload and consistency requirements rather than fashion. Important selection criteria include:

Polyglot persistence increases operational complexity. Teams must understand backup and restoration, replication, schema evolution, monitoring, access control, capacity planning, and failure modes for each technology. A service should not adopt a new database merely to avoid learning how to model data in an existing one.

A practical decision process is to identify the aggregate and access patterns first, define consistency and durability requirements, estimate scale, then compare technologies. The question is not whether a database is powerful, but whether its guarantees match the service’s responsibilities.

3. Transaction Boundaries and Consistency

A local transaction is an atomic operation within one resource manager, commonly a database. It can enforce invariants such as “an account balance cannot become negative” when the relevant records are in the same transactional boundary.

A business transaction may span several services. For example, placing an order may require reservation, payment authorization, and shipment creation. These steps may use separate databases and cannot safely be treated as one ordinary local transaction.

The familiar ACID properties are useful locally:

• Atomicity: all changes in the transaction commit, or none do.
• Consistency: declared database constraints and business invariants are preserved.
• Isolation: concurrent transactions do not observe prohibited intermediate states.
• Durability: committed changes survive the specified failure conditions.

Distributed systems often accept eventual consistency: replicas or projections may temporarily differ but converge after successful message processing. Eventual consistency is not the absence of rules. A design must specify what may be temporarily stale, what must be immediately correct, and how conflicts are resolved.

Consistency should be stated in business terms. “Inventory is eventually consistent” is incomplete. A stronger statement is: “The inventory authority never confirms more units than its available quantity, while search results may display availability up to several seconds old.”

Two-phase commit and practical alternatives

Two-phase commit coordinates participants through a prepare phase and a commit phase. It can provide atomicity across resources, but participants may hold locks or prepared state while waiting for a coordinator. Network failures, coordinator failures, and heterogeneous resource support make it costly and operationally fragile for many microservices. It is not automatically wrong, but it should be justified by a genuine need for distributed atomicity.

Sagas are often preferred when the business process can tolerate intermediate states and has meaningful recovery actions.

4. Sagas

A saga is a sequence of local transactions. Each transaction commits independently and triggers the next step. If a later step fails, the process executes compensating actions for completed steps.

A simplified order saga might be:

1. Create an order in PENDING status.
2. Reserve inventory.
3. Authorize payment.
4. Arrange shipment.
5. Mark the order CONFIRMED.

If payment fails, the system may release inventory and mark the order PAYMENT_FAILED. The process does not undo history; it records new facts that compensate for prior effects.

Choreography and orchestration

In choreography, services react to events without a central coordinator. For example, an OrderCreated event causes inventory to reserve stock; InventoryReserved causes payment to authorize; and PaymentAuthorized causes shipping to prepare delivery.

Choreography can be loosely coupled and simple for short workflows. However, complex processes can become difficult to understand because control logic is distributed among subscribers. Diagnosing missing or duplicated events also becomes harder.

In orchestration, a saga orchestrator directs participants and records workflow state. The orchestrator can make process state visible and apply timeouts, retries, and compensations consistently. It may, however, become overly powerful or tightly coupled to every service if it contains business logic that belongs elsewhere.

Compensating actions

A compensating action is a new valid business operation that offsets an earlier effect. A refund may compensate for a captured payment; releasing a reservation may compensate for inventory allocation. Compensation is not always exact reversal. A shipped item cannot be unshipped, and an email cannot be made unread.

Good saga steps should be:

• Idempotent: repeating the same request produces the same effective result.
• Observable: state transitions and failures are traceable.
• Retry-aware: transient errors are distinguished from permanent business rejection.
• Timeout-aware: a process does not wait forever.
• Explicit about human intervention: some failures require review rather than automatic reversal.

A useful state model is PENDING → COMPLETED, with explicit failure states such as REJECTED, CANCELLED, and COMPENSATION_REQUIRED. Ambiguous states such as “payment outcome unknown” should not be treated as ordinary failure; the service may need reconciliation with the payment provider.

5. Reliable Event Publication

A service often needs to update its database and publish an event. If it commits the database first and crashes before publishing, consumers never receive the event. If it publishes first and then the database transaction fails, consumers receive a fact that is not true. This is the dual-write problem.

Transactional outbox

The transactional outbox pattern stores the business change and an event record in the same local database transaction. A relay later reads unsent outbox records and publishes them to a broker.

Figure: A local transaction records both business state and an outbox event before a relay publishes the event.

The relay may publish an event more than once, so consumers must be idempotent. A consumer can store processed event identifiers, use an idempotent business operation, or rely on a broker and processing design that makes duplicates harmless. The outbox also requires retention, cleanup, monitoring, ordering decisions, and a policy for permanently failing messages.

Change-data capture

Change-data capture (CDC) observes database changes and emits them as events. Log-based CDC commonly reads a database’s transaction log, allowing changes to be captured without modifying every application write path. CDC is useful for feeding search indexes, warehouses, and integration streams.

CDC does not automatically create good domain events. A row update such as status = 3 may reveal implementation detail rather than the business meaning “PaymentAuthorized.” Teams must decide whether consumers need raw data changes, enriched integration events, or domain events. Schema evolution, deleted records, ordering, privacy, and replay behavior require explicit design.

6. CQRS and Materialized Views

Command Query Responsibility Segregation (CQRS) separates models used to change state from models used to read state. The command side validates business operations and writes authoritative data. The query side maintains representations optimized for specific read patterns.

CQRS is valuable when read and write workloads differ substantially, when a read requires data from several services, or when read models need different indexing and shapes. It introduces eventual consistency and projection-management costs, so it should not be applied to every CRUD service.

A materialized view is a stored result or projection derived from source data. For example, an order-history view may combine order, payment, and shipment events into one query-friendly document. It can be rebuilt from retained events or source records, provided the reconstruction process is reliable.

Projection design should define:

• Source events or records.
• Transformation rules and versioning.
• Ordering and duplicate handling.
• Rebuild and backfill procedures.
• Behavior when source data is unavailable.
• Whether the view is authoritative or merely a convenience.

7. Event Sourcing

Event sourcing stores a sequence of domain events as the primary record of state. Current state is reconstructed by replaying events, often with snapshots to reduce replay cost. An event such as OrderPlaced describes a business fact, whereas a mutable row may only show the latest status.

Benefits include an auditable history, temporal analysis, and the ability to build new projections. Costs include event-schema evolution, storage growth, replay complexity, privacy management, and the difficulty of correcting a published fact. Event sourcing is not the same as ordinary event publication: a system can publish events from a conventional state store, and an event-sourced system still needs projections for efficient queries.

Events should be immutable facts. Corrections are represented by new events, such as AddressCorrectionRecorded, rather than rewriting history without trace. Sensitive data must be handled carefully because immutable events complicate deletion and rectification obligations.

8. Caching and Distributed Data

A cache stores data closer to consumers to reduce latency or load. Common patterns include:

• Cache-aside: the application reads the cache, loads from the source on a miss, and writes the result to the cache.
• Read-through: the cache itself loads missing data through a configured loader.
• Write-through: writes pass through the cache to the authoritative store.
• Write-behind: the cache acknowledges writes before asynchronously persisting them; this requires strong recovery controls.

Cache-aside is common, but invalidation is difficult. A time-to-live (TTL) limits staleness but does not guarantee freshness before expiry. If a value is cached at time t₀ and read at time t₁, its age is t₁ − t₀; this is a time difference, not a guarantee that the underlying value has not changed. Event-driven invalidation can reduce staleness but introduces another delivery dependency.

Never cache authorization decisions or sensitive data without considering identity, tenant isolation, encryption, and revocation. Cache keys must include all dimensions that affect the result, such as tenant, locale, and permission scope. A cache failure should normally degrade performance rather than silently produce unsafe business decisions.

9. Illustrative Case: An Online Retail Order

Background

A fictional retailer operates catalog, order, inventory, payment, and shipping services. Each service has its own database. Customers expect rapid browsing, while order confirmation must not promise unavailable inventory.

Problem

The original design used a shared database transaction across all services. It became difficult to deploy services independently, and a payment-provider timeout held database locks. The team also found that search results and inventory quantities were frequently stale.

Analysis

The business distinguishes authoritative facts from convenient projections. Catalog owns product descriptions and prices. Inventory owns available stock. Payment owns authorization status. Order owns the commercial order and its lifecycle. Search owns a materialized index and is allowed to be eventually consistent.

Approach

An order request creates an order in PENDING status and an outbox event in one local transaction. A saga orchestrator requests inventory reservation. Inventory performs a local transaction that atomically checks and reserves stock, then publishes InventoryReserved or InventoryRejected. If reserved, the orchestrator requests payment authorization. If payment is rejected, it requests release of the reservation and marks the order failed. If payment is authorized, shipping is requested and the order eventually becomes confirmed.

Each request contains a unique operation identifier. Inventory and payment record processed identifiers so retries do not create duplicate reservations or charges. A reconciliation job compares uncertain payment states with the provider. Search is updated through events and may lag by several seconds. A read model combines order, payment, and shipment statuses for customer support.

Outcome

The retailer removes cross-service database writes, limits long-held locks, and gains independently scalable search and checkout components. The system accepts that customer-facing projections may be temporarily stale, while inventory reservation and payment rules remain authoritative within their respective services.

Learning points

1. Local atomicity is preserved inside each service.
2. The saga expresses business progress without requiring one global transaction.
3. Compensation changes business state; it does not erase history.
4. Idempotency and reconciliation are essential, not optional enhancements.
5. Read models and caches improve performance but do not become authorities accidentally.

10. Governance, Security, and Professional Practice

Distributed data increases the number of places where personal and commercially sensitive information may exist. Data classification should identify personal, financial, confidential, and regulated fields. Teams should minimize replication, encrypt data in transit and at rest, restrict service credentials, rotate secrets, and log access without exposing sensitive payloads.

Retention and deletion are especially challenging with backups, event logs, outboxes, caches, and analytical copies. A deletion policy must define how each representation is removed, anonymized, or rendered inaccessible. Legal requirements vary by jurisdiction; technical teams should work with privacy and compliance specialists rather than assuming that an event log is exempt from deletion obligations.

Observability should include correlation identifiers, event identifiers, workflow state, retry counts, age of outbox records, projection lag, cache hit ratio, and reconciliation exceptions. Distributed tracing can show the path of a request, but traces are not a substitute for durable business history.

AI-assisted development can generate schemas, migration scripts, and event handlers, but generated code must be reviewed for transaction boundaries, data leakage, authorization, idempotency, and failure behavior. An AI tool should not receive production personal data merely to diagnose a query.

11. Best-Practice Design Checklist

Before implementing distributed data management, ask:

• Is every important fact assigned to one clear owner?
• Are cross-service writes prohibited or explicitly governed?
• Which invariants must be immediate, and which may be eventual?
• Can each message be duplicated, delayed, reordered, or lost?
• Are handlers idempotent and are failures observable?
• Does every saga have timeouts, compensation, and reconciliation paths?
• Is the outbox or CDC pipeline monitored and replayable?
• Can projections be rebuilt from defined sources?
• Are cache staleness and invalidation acceptable for the business use case?
• Are privacy, retention, access control, and audit requirements satisfied?

Chapter Summary

Distributed data management begins with clear data ownership. Database-per-service protects autonomy when services own their schemas and expose contracts rather than shared tables. Polyglot persistence can match storage technology to workload, but it increases operational responsibility.

Local transactions are effective within a service, whereas cross-service business processes generally require sagas, explicit state transitions, idempotency, retries, timeouts, and compensating actions. The transactional outbox addresses the dual-write problem by recording business changes and events atomically. CDC can distribute database changes efficiently, although raw row changes are not automatically meaningful domain events.

CQRS separates command and query responsibilities, while materialized views provide efficient read representations. Event sourcing preserves an event history but introduces replay, schema, privacy, and correction challenges. Caches improve performance but create freshness and invalidation concerns. Across all patterns, authoritative ownership, observability, security, governance, and recovery design determine whether a distributed data architecture is dependable.

Key Terms

Data ownership: Responsibility for defining, validating, changing, and correcting a business fact.

Database-per-service: A design in which each service controls its own data store or logical database boundary.

Polyglot persistence: Use of multiple data-storage technologies selected for different requirements.

Bounded context: A domain boundary within which a model and its terminology have consistent meaning.

Local transaction: An atomic transaction managed within one service and its resource manager.

Eventual consistency: A model in which replicas or projections converge after successful updates.

Saga: A sequence of local transactions coordinated through messages and compensating actions.

Choreography: Saga coordination achieved through events and autonomous service reactions.

Orchestration: Saga coordination directed by a workflow coordinator.

Compensating action: A new business operation that offsets the effect of an earlier operation.

Idempotency: The property that repeating an operation does not create an additional effective outcome.

Transactional outbox: A table or store recording events in the same local transaction as business data.

Change-data capture: A technique for detecting and publishing changes from a data source.

CQRS: Separation of models or pathways for commands and queries.

Materialized view: A stored, derived representation optimized for reading.

Event sourcing: Persistence of domain events as the primary record from which state is reconstructed.

Cache-aside: A caching pattern in which the application loads missing values and manages cache writes.

Reconciliation: A process that compares systems or providers to resolve uncertain or divergent states.

Review Questions

Short Answer Questions

1. What does data ownership mean in a microservices architecture?
2. Why can direct cross-service database access create coupling?
3. Define a saga and a compensating action.
4. What problem does the transactional outbox address?
5. Why can a cache not normally be treated as the system of record?

Long Answer Questions

1. Compare database-per-service and a shared database architecture, including benefits, risks, and appropriate use cases.
2. Explain choreography-based and orchestration-based sagas and evaluate their relative strengths.
3. Describe CQRS, materialized views, and event sourcing, including the conditions under which each is useful.
4. Explain how idempotency, retries, timeouts, and reconciliation contribute to reliable distributed workflows.
5. Discuss the security, privacy, retention, and governance challenges of distributed data.

Analytical / Application Questions

1. Design a saga for a hotel booking involving room allocation, payment authorization, and confirmation. Identify failure states and compensating actions.
2. A service updates its database successfully but crashes before publishing an event. Analyze the consequences and propose a reliable publication design.
3. A customer dashboard combines data from orders, payments, and shipments. Recommend an ownership model, projection strategy, cache policy, and consistency guarantees, explaining your trade-offs.

Multiple Choice Questions with Answers

Question 1

Which statement best describes data ownership in a microservices architecture?

A. Every service may directly update any shared table.

B. One service is accountable for validating and changing an authoritative business fact.

C. The cache becomes authoritative because it is fastest.

D. Ownership is assigned to the service with the largest database.

Correct Answer: B. One service is accountable for validating and changing an authoritative business fact.

Question 2

What is the primary purpose of a transactional outbox?

A. To eliminate the need for database transactions.

B. To guarantee that a message is delivered exactly once.

C. To record business data and an event atomically before publication.

D. To replace all message brokers with a relational database.

Correct Answer: C. To record business data and an event atomically before publication.

Question 3

A saga compensating action is best understood as:

A. A rollback of every physical effect through a global transaction.

B. A new business operation that offsets an earlier effect.

C. A database backup created before a transaction.

D. A cache refresh performed after every read.

Correct Answer: B. A new business operation that offsets an earlier effect.

Question 4

Which property is most important when a message may be delivered more than once?

A. Idempotency

B. Randomization

C. Denormalization

D. Compression

Correct Answer: A. Idempotency

Question 5

What is a principal characteristic of CQRS?

A. It requires every service to use a graph database.

B. It separates models or paths used for commands and queries.

C. It prevents all eventual consistency.

D. It replaces domain events with database locks.

Correct Answer: B. It separates models or paths used for commands and queries.

Question 6

Which statement about change-data capture is correct?

A. It always produces meaningful domain events automatically.

B. It observes changes in a data source and can publish them to downstream consumers.

C. It requires all services to share one schema.

D. It guarantees that consumers never see duplicate changes.

Correct Answer: B. It observes changes in a data source and can publish them to downstream consumers.

Question 7

Why might event sourcing create privacy challenges?

A. Events cannot be stored on durable media.

B. Immutable historical events can make deletion or rectification more difficult.

C. Event-sourced systems cannot use encryption.

D. Events are always stored in browser caches.

Correct Answer: B. Immutable historical events can make deletion or rectification more difficult.

Question 8

Which service should be authoritative for available inventory in the retail case?

A. Search service

B. Order-history read model

C. Inventory service

D. Customer dashboard cache

Correct Answer: C. Inventory service

Question 9

What is a likely consequence of using a cache-aside pattern?

A. The cache can contain stale data until invalidation or expiration.

B. The source database is never queried.

C. All writes become globally atomic.

D. Duplicate messages become impossible.

Correct Answer: A. The cache can contain stale data until invalidation or expiration.

Question 10

A materialized view is primarily:

A. A stored representation derived and optimized for query access.

B. A global transaction coordinator.

C. A mandatory replacement for an authoritative database.

D. A mechanism that prevents schema evolution.

Correct Answer: A. A stored representation derived and optimized for query access.