Data Management for Distributed Services

Introduction

Microservices and serverless systems distribute computation across independently deployable services, functions, regions, and infrastructure providers. This distribution creates a corresponding data-management problem: data is no longer naturally located in one database under one transaction boundary. A service may own customer data, another may own orders, and a third may maintain payment records. Each service must remain autonomous while the overall system still provides useful business behaviour.

Data management for distributed services therefore concerns more than choosing a database product. It involves deciding who owns data, how services communicate, how information is replicated, how consistency is defined, how transactions cross service boundaries, how schemas evolve, and how data is protected and recovered. Poor decisions produce hidden coupling, contradictory records, unavailable services, privacy violations, and difficult operational recovery.

This chapter develops practical design principles for database-per-service architecture, polyglot persistence, caching, distributed transactions, schema evolution, replication, consistency, privacy, and disaster recovery.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain data ownership and the database-per-service pattern.
2. Select storage technologies using polyglot persistence principles.
3. Distinguish strong, eventual, causal, and session consistency.
4. Design caching strategies and evaluate their failure modes.
5. Compare sagas, transactional outbox, two-phase commit, and compensating actions.
6. Plan schema evolution for APIs, events, and databases.
7. Analyse replication, recovery objectives, privacy, and governance requirements.
8. Apply these concepts to a realistic distributed-service design.

1. Data Ownership and the Database-per-Service Principle

A service owns a data set when it is responsible for its meaning, validation rules, lifecycle, security policy, and authoritative updates. Ownership is a business and architectural responsibility, not merely permission to connect to a database.

In a database-per-service architecture, each service has exclusive control over its persistent storage. “Database per service” does not necessarily mean one physical database server per service. It means that other services do not directly read or modify the owner’s tables. A service may use a separate database, schema, account, or logically isolated storage system, provided the ownership boundary is enforceable.

For example, an online retailer might define:

A service may expose data through an API or publish domain events. Other services maintain local projections when they need efficient queries. They should not bypass the owner by joining directly to its tables.

Why ownership matters

Direct cross-service database access creates temporal, structural, and semantic coupling. A schema change in one service can unexpectedly break another. A query may depend on a transaction that no longer exists. Developers may interpret the same field differently. Operationally, one service can overload another service’s database without passing through an observable API contract.

Ownership also clarifies authority. If both Order and Payment can change a payment status, it becomes difficult to determine which value is correct. A better design makes Payment authoritative for payment state and informs Order through an event such as PaymentAuthorized or PaymentDeclined.

Shared database: when it may be acceptable

A shared database can be reasonable during an incremental migration, in a small monolith, or when several modules genuinely form one transaction boundary. It becomes risky when independently deployable services use shared tables as an undocumented integration mechanism. If a shared database is retained, ownership, access rights, migration responsibility, and deprecation plans should be explicit.

2. Polyglot Persistence

Polyglot persistence means selecting storage technologies according to workload and data-model requirements rather than imposing one database technology on every service. It is not a justification for technological variety without governance.

Common categories include:

Technology selection should begin with access patterns: read/write ratio, latency target, consistency requirement, record size, query shape, retention period, regulatory classification, and expected growth. A database should not be selected merely because it is popular or managed by a cloud provider.

Polyglot persistence governance

Each additional technology increases operational and skills complexity. A professional platform should standardise monitoring, backup testing, identity integration, encryption, patching, schema migration, cost reporting, and incident procedures. Teams should document the reason for each technology and identify the authoritative source of every important business fact.

3. Consistency Models and Replication

Replication maintains multiple copies of data for availability, performance, geographic proximity, or disaster recovery. Replication does not automatically make a system more reliable: it introduces lag, conflict, failover decisions, and possible divergence.

Consistency models

Strong consistency means that a successful read reflects the latest completed write according to the system’s defined ordering. It is useful for operations such as preventing duplicate financial settlement, although it may reduce availability or increase latency across regions.

Eventual consistency means that, if no new updates occur and communication continues to function, replicas converge to the same value. During convergence, different clients may observe different values. This is often acceptable for search indexes, recommendation data, and social activity feeds.

Causal consistency preserves cause-and-effect ordering. If an update is observed before a related update is issued, another reader will not observe the effect without its cause. This is useful for user-visible workflows involving related events.

Session guarantees provide practical client-oriented properties such as read-your-writes, monotonic reads, and monotonic writes. They can improve user experience without requiring global strong consistency.

Quorum reasoning

Some replicated systems use read and write quorums. Let N be the number of replicas, W the number that must acknowledge a write, and R the number consulted by a read. Under simplified assumptions of a single replica group, reliable replica membership, and suitable conflict handling, a read and write quorum overlap when:

This overlap can help a read encounter a replica containing the latest acknowledged write. The relationship alone does not guarantee linearizability, because implementation details, concurrent writes, clocks, leader changes, and conflict resolution also matter. For N = 3, W = 2 and R = 2 satisfy the overlap condition because 2 + 2 > 3.

Replication choices

Synchronous replication waits for specified replicas before acknowledging a write. It can reduce acknowledged-data loss but may increase latency and reduce availability when replicas are unreachable. Asynchronous replication acknowledges before remote copies are confirmed. It generally improves latency and availability but creates a recovery point during which recent writes may be lost.

Leader-follower replication directs writes to a leader and copies them to followers. Multi-leader replication permits writes in multiple locations but requires conflict detection and resolution. Leaderless designs commonly use quorum techniques and versioning. The correct choice depends on conflict frequency, geographic requirements, and business tolerance for stale or conflicting data.

4. Caching Distributed Data

A cache stores data closer to the consumer or in a faster medium. A cache is normally a performance optimisation, not the authoritative source of business truth. Treating a cache as durable truth without appropriate persistence and recovery controls is a design error.

Common patterns include:

• Cache-aside: the application reads the cache; on a miss, it reads the database and populates the cache. The application invalidates or refreshes entries after writes.
• Read-through: the cache itself loads missing data from the backing store.
• Write-through: a write updates the cache and backing store synchronously.
• Write-behind: the cache acknowledges first and writes later; this improves latency but risks data loss and ordering problems.
• Refresh-ahead: frequently accessed entries are refreshed before expiry.

Cache-aside is widely used because it is simple and keeps the database authoritative. Its principal challenge is stale data between a database update and cache invalidation. A time-to-live (TTL) limits staleness but does not eliminate it. Short TTLs reduce stale exposure but increase database load; long TTLs improve hit rate but may serve older values.

A useful operational measure is cache hit ratio:

The ratio is dimensionless and should be measured for a defined time interval and workload. A high hit ratio does not prove correctness: a cache can return stale or unauthorised data efficiently.

Important cache risks include cache stampede, where many requests reload an expired item simultaneously; cache penetration, where repeated requests for nonexistent keys reach the database; and cache poisoning, where incorrect or unauthorised data is inserted. Mitigations include request coalescing, jittered expiry, negative caching with careful TTLs, rate limiting, validation, and namespace isolation.

Personal data requires special care. Cache keys and values must respect tenant boundaries, authorisation context, retention rules, and deletion requests. Sensitive records should not be cached by shared intermediaries unless encryption, access control, and invalidation are demonstrably correct.

5. Distributed Transactions

A local database transaction can provide atomicity within one transaction manager. A business process spanning several services cannot automatically obtain the same guarantee merely because each service uses transactions locally. Distributed transaction design must instead define acceptable intermediate states and recovery behaviour.

Two-phase commit

Two-phase commit (2PC) uses a coordinator. In the prepare phase, participants record that they can commit. In the commit phase, the coordinator instructs all participants to commit. 2PC can provide atomic commitment under its model, but participants may hold locks or prepared state while the coordinator is unavailable. It can therefore reduce availability and increase latency. It is generally unsuitable for loosely coupled, geographically distributed microservices unless the infrastructure and workload explicitly justify it.

Saga pattern

A saga divides a business transaction into a sequence of local transactions. Each successful step publishes an event or triggers the next step. If a later step fails, compensating actions attempt to reverse the business effect of earlier steps.

For an order process:

1. Order service creates an order in PENDING state.
2. Inventory service reserves stock.
3. Payment service authorises payment.
4. Delivery service creates a shipment.
5. Order service marks the order CONFIRMED.

If payment fails, inventory can release its reservation and Order can mark the order REJECTED. Compensation is not always a perfect inverse. A refund may be possible after payment capture, but a notification already sent cannot be unsent. The domain must therefore model states such as RESERVED, CAPTURED, RELEASED, REFUNDED, and FAILED rather than pretending that rollback is universal.

Sagas may be orchestrated by a central workflow component or coordinated through events. Orchestration makes process control visible; choreography reduces central coordination but can become difficult to understand as event chains grow.

Transactional outbox

The transactional outbox solves a common dual-write problem: a service must update its database and publish an event. The service writes both the business change and an outbox record in one local transaction. A relay later publishes the outbox event. Consumers must be idempotent because publication may be retried and the same event may be delivered more than once.

An event should have a stable identifier, event type, aggregate identifier, version, creation time, and sufficient payload or reference for the consumer. The outbox requires monitoring for backlog, failed delivery, poison messages, and retention.

Idempotency

An operation is idempotent if repeating it has the same intended business effect as performing it once. Payment APIs commonly accept an idempotency key. The service stores the key and result, then returns the original result for a retry. Idempotency is essential because timeouts do not reveal whether the server processed a request before the client lost the response.

6. Schema Evolution and Contract Management

Distributed systems cannot assume that all consumers upgrade simultaneously. Schema evolution must therefore support compatibility during rolling deployment and delayed consumer upgrades.

For APIs, additive changes are usually safer than destructive changes. Adding an optional response field is generally compatible; renaming or removing a required field is not. A producer should tolerate unknown fields where appropriate, and consumers should avoid failing merely because a new field appears.

For events, event names and meanings are contracts. A field should not silently change from “amount in cents” to “amount in dollars.” Units, nullability, enumerated values, ordering, ownership, and version policy should be documented. Schema registries can validate event structure, but they do not guarantee semantic compatibility.

Database migration patterns include:

1. Expand: add the new column or structure while retaining the old one.
2. Migrate: write new data and backfill existing records.
3. Dual-read or dual-write temporarily: compare results and detect defects.
4. Contract: stop using the old structure after all consumers migrate.
5. Remove: delete obsolete fields only after a verified deprecation period.

Large backfills should be throttled, resumable, observable, and tested against production-like data. Destructive migrations require backups and a tested rollback or forward-recovery plan. “Rollback” may be unsafe after irreversible business actions; restoring a database snapshot does not undo an email, shipment, or payment already processed.

7. Privacy, Security, and Governance

Distributed data increases the number of copies, logs, queues, caches, backups, indexes, and replicas that may contain personal or confidential information. Privacy must therefore be designed across the complete data lifecycle.

Key controls include:

• data minimisation: collect and copy only what is necessary;
• purpose limitation: use data for authorised purposes;
• classification: identify public, internal, confidential, and sensitive data;
• encryption in transit and at rest, with controlled key management;
• least-privilege identities for services and operators;
• tenant isolation and object-level authorisation;
• audit trails for access and administrative actions;
• retention and deletion policies applied to primary and derived stores;
• masking or tokenisation in testing and observability systems;
• documented data-processing locations and transfer restrictions.

A deletion request is not complete if the primary record is deleted but an unexpired cache, search index, event replay store, or backup still exposes the data. Some systems must retain legally required records, so retention and erasure obligations require legal and compliance interpretation rather than purely technical assumptions.

Logs should avoid secrets, authentication tokens, full payment data, and unnecessary personal information. Correlation identifiers are useful, but they should not themselves reveal sensitive attributes.

8. Disaster Recovery and Operational Resilience

Disaster recovery addresses how a service restores operation and data after hardware failure, accidental deletion, software defects, ransomware, regional outage, or provider failure.

Two important objectives are:

• Recovery Point Objective (RPO): the maximum acceptable amount of data loss measured in time. An RPO of 15 minutes means the organisation aims to lose no more than approximately 15 minutes of acknowledged data under the defined disaster scenario.
• Recovery Time Objective (RTO): the target maximum time to restore service after disruption.

These objectives are business requirements, not merely database settings. A zero-RPO design may require synchronous replication or another confirmed-durability mechanism and may have significant cost and latency implications.

Backups should be automated, encrypted, access-controlled, versioned, and isolated from ordinary production credentials. A backup that has never been restored is an assumption, not evidence. Recovery tests should verify data integrity, application compatibility, permissions, replay of events, DNS or routing changes, and customer-facing behaviour.

A practical recovery plan identifies dependencies and ordering. For example, identity and key-management services may need to recover before databases, and databases before application services. Runbooks should state who declares a disaster, who approves failover, how replication lag is assessed, and how the system prevents split-brain operation.

9. A Decision Framework

A data design can be evaluated through the following sequence:

1. Identify business aggregates and authoritative owners.
2. List commands, queries, events, and access patterns.
3. Define consistency requirements for each operation, not for the system as a whole.
4. Select storage technology based on workload, compliance, and operational capability.
5. Decide which data can be cached or materialised.
6. Identify workflows crossing service boundaries and choose saga, outbox, or another mechanism.
7. Define schema compatibility and migration policy.
8. Specify replication topology, RPO, RTO, and failover behaviour.
9. Classify data and apply privacy controls to every copy.
10. Test failure, duplication, delay, stale reads, partial completion, and recovery.

This process prevents the common error of selecting a database before understanding the business invariants.

Applied Case: Illustrative Online Retail Platform

Background

An online retailer is decomposing a monolithic application into Order, Inventory, Payment, and Delivery services. It operates in two geographic regions and experiences high read traffic for product availability.

Problem

The original system uses shared tables. During sales campaigns, product pages overload the order database. A payment timeout occasionally causes customers to retry, producing duplicate charges. Deployments are delayed because several modules depend on the same schema.

Analysis

The retailer identifies Order as the owner of order lifecycle, Inventory as the owner of stock reservations, and Payment as the owner of payment state. Product availability is a read-oriented projection and does not need to be the authoritative stock ledger. Payment and inventory operations require idempotency because clients and message brokers may retry.

Approach

Each service receives an isolated data store. Order uses a relational database for transactional order aggregates. Inventory uses a store optimised for reservation lookups and maintains a durable reservation record. Payment uses a relational store with encrypted sensitive references and an idempotency-key table. A search index and cache serve product queries but are rebuilt from authoritative events when necessary.

The checkout process is implemented as an orchestrated saga. Order creates PENDING. Inventory reserves stock. Payment authorises funds. On failure, the workflow releases stock or requests a refund where appropriate. The transactional outbox publishes OrderCreated, StockReserved, PaymentAuthorized, and related events. Consumers record processed event identifiers so retries are safe.

The platform uses cache-aside for product information with a bounded TTL and explicit invalidation after catalogue changes. It does not cache payment credentials. Cross-region replicas are asynchronous for catalogue and search data, while payment writes use a stronger durability configuration. Backups are encrypted and restoration is tested quarterly.

Outcome

Product reads no longer compete directly with order writes. Deployment teams can evolve service schemas independently. Duplicate payment attempts return the stored result for the same idempotency key. The retailer accepts that availability projections may be briefly stale, but checkout validates reservation state against the authoritative Inventory service.

Learning points

The case demonstrates that consistency should be attached to business decisions. Stale search information may be acceptable, but approving a payment against an invalid reservation is not. It also shows that a saga coordinates business state without pretending that distributed rollback is identical to a local database rollback.

Challenges and Limitations

Distributed data management increases operational complexity. Eventual consistency can confuse users and support staff when screens disagree temporarily. Sagas create intermediate states and require durable workflow history. Polyglot persistence increases platform skills and monitoring requirements. Replication can produce conflicts or stale reads. Caches can hide defects until expiry or failover. Backups may preserve corrupted data if corruption is replicated before detection.

No architectural pattern removes the need for domain modelling, observability, testing, and clear ownership. The appropriate design is the simplest one that satisfies the required business, performance, resilience, and regulatory constraints.

Contemporary Developments

Managed cloud databases increasingly provide serverless scaling, change streams, global tables, and automated backups. These capabilities reduce infrastructure administration but do not remove questions of cost, consistency, quotas, portability, and data residency. Event-driven architectures and workflow engines make long-running sagas more observable. Confidential-computing technologies and tokenisation can reduce exposure of sensitive data, although their assurance depends on implementation and governance.

Artificial intelligence is relevant when services store embeddings, prompts, model outputs, or training data. Such data requires provenance, retention controls, access restrictions, evaluation records, and protection against personal-data leakage. An embedding index should not be assumed to be harmless merely because it is not human-readable.

Best Practices

• Make one service authoritative for each business fact.
• Prefer APIs and events over shared-table access.
• Choose consistency per operation and business invariant.
• Use idempotency for externally retried commands.
• Use transactional outbox for reliable local change publication.
• Design sagas with explicit states, timeouts, retries, and compensations.
• Treat caches and search indexes as reconstructible unless deliberately made authoritative.
• Use expand-and-contract schema migration.
• Test replication lag, duplicate delivery, partial failure, and restore procedures.
• Apply privacy controls to replicas, logs, queues, caches, indexes, and backups.
• Measure recovery objectives through exercises rather than documentation alone.

Chapter Summary

Distributed services require explicit data ownership because independent deployment is incompatible with uncontrolled shared-table dependencies. Database-per-service establishes ownership, while APIs, events, and local projections provide integration. Polyglot persistence matches storage technology to workload but requires operational governance.

Replication improves availability and locality while introducing lag and conflict concerns. Strong, eventual, causal, and session consistency are different guarantees, and each operation should receive the weakest model that safely supports its business invariant. Caching improves performance but must address staleness, stampedes, privacy, and invalidation.

Distributed business processes are usually implemented with local transactions, transactional outboxes, idempotent consumers, and sagas rather than global rollback. Schema evolution requires compatibility and expand-and-contract migration. Privacy applies to every copy and derivative of data. Disaster recovery depends on defined RPO and RTO, isolated tested backups, dependency-aware failover, and regular exercises.

Key Terms

Data ownership: Responsibility for the meaning, validation, lifecycle, and authoritative modification of data.

Database per service: A design in which a service exclusively controls its persistent data boundary.

Polyglot persistence: Use of different storage technologies selected for different workload requirements.

Strong consistency: A guarantee that reads reflect the latest completed write under the system’s ordering model.

Eventual consistency: A model in which replicas converge when updates cease and communication continues.

Causal consistency: Preservation of cause-and-effect ordering between related operations.

Replication: Maintaining multiple copies of data.

Quorum: A sufficient subset of replicas required for a read or write operation.

Cache-aside: A pattern in which the application loads missing data into a cache and manages invalidation.

Time-to-live (TTL): The period after which a cached item expires.

Saga: A sequence of local transactions coordinated with events or a workflow and compensating actions.

Compensating action: A business operation that counteracts the effect of an earlier completed operation.

Two-phase commit: A coordinated distributed commit protocol with prepare and commit phases.

Transactional outbox: A pattern that stores a business change and its outbound event in one local transaction.

Idempotency: The property that repeating an operation produces the same intended business effect.

Schema evolution: Controlled change to data structures and contracts over time.

RPO: Recovery Point Objective; the maximum acceptable time-based amount of data loss.

RTO: Recovery Time Objective; the target time for restoring service after disruption.

Data minimisation: Collecting, retaining, and copying only data necessary for a legitimate purpose.

Split-brain: A failure condition in which multiple nodes independently act as authoritative leaders.

Review Questions

Short Answer Questions

1. What does data ownership mean in a microservice architecture?
2. Distinguish strong consistency from eventual consistency.
3. Why is idempotency important when processing payment requests?
4. What problem does the transactional outbox pattern address?
5. Define RPO and RTO.

Long Answer Questions

1. Explain the benefits and risks of database-per-service architecture.
2. Compare cache-aside, write-through, and write-behind caching.
3. Explain how a saga coordinates a business process across services.
4. Describe an expand-and-contract approach to database schema evolution.
5. Discuss privacy risks created by replicas, caches, logs, and backups.

Analytical / Application Questions

1. Design a data architecture for a food-delivery platform. Identify service owners, storage technologies, consistency requirements, and events.
2. A replicated system has N = 5 replicas, W = 3, and R = 2. Determine whether the simplified quorum-overlap condition is satisfied, and explain why the result does not by itself guarantee linearizability.
3. An order workflow successfully reserves stock but times out during payment authorisation. Propose states, retry rules, idempotency controls, and compensating actions that prevent duplicate charging and incorrect inventory release.

Multiple Choice Questions with Answers

Question 1

Which principle best describes database-per-service architecture?

A. Every service directly joins tables in a central database

B. Each service exclusively controls its persistent data boundary

C. All services must use the same database technology

D. Only one service is allowed to store data

Correct Answer: B. Each service exclusively controls its persistent data boundary

Question 2

Which storage type is generally most appropriate for full-text search and ranking?

A. Search engine

B. Object storage

C. Key-value store

D. In-memory cache

Correct Answer: A. Search engine

Question 3

For N = 3 replicas, which pair satisfies the simplified quorum-overlap condition R + W > N?

A. R = 1, W = 1

B. R = 1, W = 2

C. R = 2, W = 2

D. R = 1, W = 0

Correct Answer: C. R = 2, W = 2

Question 4

In the cache-aside pattern, what normally happens after a cache miss?

A. The application reads the backing store and populates the cache

B. The cache permanently deletes the database record

C. The database waits for every cache replica to expire

D. The application writes only to the cache

Correct Answer: A. The application reads the backing store and populates the cache

Question 5

What is the main purpose of a transactional outbox?

A. To guarantee that a cache never becomes stale

B. To atomically record a local business change and an event for later publication

C. To replace all database backups

D. To force every service to use two-phase commit

Correct Answer: B. To atomically record a local business change and an event for later publication

Question 6

Which statement about saga compensation is correct?

A. Every business action can always be perfectly rolled back

B. Compensation is a business operation that attempts to counteract an earlier action

C. Compensation requires deleting all event records

D. A saga eliminates the need for idempotency

Correct Answer: B. Compensation is a business operation that attempts to counteract an earlier action

Question 7

Which schema change is usually most compatible during rolling deployment?

A. Removing a required field immediately

B. Changing the unit of an existing amount without notice

C. Adding an optional field while retaining existing fields

D. Renaming an event without notifying consumers

Correct Answer: C. Adding an optional field while retaining existing fields

Question 8

What does an RPO of 15 minutes primarily describe?

A. The maximum target time to restart a server

B. The maximum acceptable time-based amount of data loss

C. The number of replicas required

D. The cache expiration interval

Correct Answer: B. The maximum acceptable time-based amount of data loss

Question 9

Which is a major privacy risk in distributed data systems?

A. A query uses an index

B. Personal data remains in logs, caches, replicas, or backups after deletion from the primary store

C. A service uses an API

D. A database has a documented owner

Correct Answer: B. Personal data remains in logs, caches, replicas, or backups after deletion from the primary store

Question 10

Why are idempotency keys useful for payment operations?

A. They compress payment records

B. They ensure retries return or reproduce one intended business result rather than creating duplicate charges

C. They make all payment data public

D. They remove the need for authentication

Correct Answer: B. They ensure retries return or reproduce one intended business result rather than creating duplicate charges