Data Management and Distributed Transactions

Introduction

Serverless applications and microservices rarely share one database. Instead, each service usually owns the data required for its business capability and exposes that data through an API or events. This arrangement improves autonomy and scalability, but it also removes the convenient single-database transaction. A business operation such as placing an order may update inventory, payment, delivery, and notification data in different systems and at different times.

This chapter explains how to select data services and design reliable distributed workflows. It examines relational, NoSQL, object, cache, and streaming services; data ownership; replication and consistency; idempotency; Saga coordination; CQRS; and event sourcing. The central design principle is that data technology should follow access patterns, integrity requirements, scale, and operational capability—not fashion.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish major categories of cloud data services;
• assign data ownership appropriately in a microservice architecture;
• explain replication, consistency, and the CAP trade-off;
• design idempotent operations and safe retries;
• compare orchestration and choreography in Saga workflows;
• explain CQRS and event sourcing and identify their limitations;
• select data services for a practical serverless application; and
• evaluate governance, security, and operational risks in distributed data systems.

1. Data as a Distributed-System Concern

In a traditional monolithic application, several tables may be updated within one database transaction. The database can enforce atomicity, consistency, isolation, and durability—the ACID properties. In a microservice system, however, a request may cross service boundaries, databases, queues, and third-party APIs. No single local transaction can automatically make all participants commit or roll back together.

A useful distinction is between local consistency and business consistency. Local consistency means that one data store obeys its own constraints. Business consistency means that the overall business rule eventually becomes true across services. For example, a payment service can atomically record a payment attempt, while the order service eventually changes an order from PENDING_PAYMENT to PAID after receiving a confirmed event.

Distributed systems commonly use asynchronous messaging. This improves resilience and decoupling, but introduces delay, duplicate delivery, out-of-order events, and temporary disagreement between replicas. These are not merely programming inconveniences; they are architectural properties that must be handled explicitly.

2. Selecting Data Services

2.1 Relational data services

Relational databases store structured data in tables connected by relationships. They support schemas, primary and foreign keys, constraints, indexes, joins, and transactions. They are appropriate when:

• relationships and integrity rules are important;
• transactions affect several records;
• reporting requires flexible queries and joins; or
• the data model is stable enough to justify a defined schema.

Examples include managed PostgreSQL, MySQL, and distributed SQL services. A relational service is often suitable for orders, invoices, account balances, and reservations. Serverless applications may connect through a connection pooler, proxy, or data API because opening a database connection per short-lived function can exhaust database connection limits.

Relational databases are not automatically unsuitable for high scale. Read replicas, partitioning, caching, indexing, and distributed SQL can extend their usefulness. Nevertheless, cross-region writes and globally distributed transactions may increase latency and operational complexity.

2.2 NoSQL data services

NoSQL is a broad category rather than one technology. Important models include:

NoSQL design begins with access patterns. A key-value service may be excellent when the application knows the key but poor for arbitrary joins. A document store can retrieve an aggregate efficiently, but embedding large or frequently changing subdocuments may create update and size problems. Partition keys must distribute traffic; a “hot partition” can become a bottleneck even when the overall database has spare capacity.

2.3 Object storage

Object storage keeps immutable or replaceable objects—such as images, video, backups, documents, data-lake files, and build artifacts—under keys in buckets or containers. It provides high durability and cost-effective scale, but it is not a replacement for a transactional database. Applications commonly store the binary object in object storage and retain metadata, ownership, status, and permissions in a database.

Good practice includes private-by-default buckets, encryption, lifecycle policies, versioning where useful, checksums, and short-lived signed URLs. Object names should avoid accidental exposure of personal information. Large uploads can use multipart or direct browser-to-storage methods so that a serverless function does not become a bandwidth intermediary.

2.4 Caches

A cache stores data closer to the application to reduce latency and database load. It may contain computed results, sessions, tokens, or frequently read reference data. A cache is normally a performance optimization, not the authoritative source of truth.

The principal design issue is invalidation. Common strategies include time-to-live expiration, explicit deletion after a write, write-through updates, and cache-aside loading. In cache-aside design, the application reads the cache, loads from the database on a miss, and then stores the result. Race conditions can cause a “cache stampede,” in which many requests simultaneously recompute the same missing value. Jittered expiration, request coalescing, and bounded refresh work can reduce this risk.

A cache must not silently return data beyond its acceptable staleness. For authorization, account balances, and safety decisions, stale cached data may be unacceptable unless the design explicitly accounts for it.

2.5 Streaming and event services

Streaming services retain ordered records within partitions or subjects and allow one or more consumers to process them. They are useful for integration events, audit trails, analytics, fraud detection, and asynchronous workflows. A stream is not simply a queue: messages may be retained and reread, and independent consumer groups may process the same event for different purposes.

Ordering is generally guaranteed only within a defined scope, such as a partition or key. If all events for an order must be ordered, the order identifier should normally determine their partitioning key. Consumers must still tolerate duplicates and retries. A durable stream does not guarantee that every downstream side effect is exactly once.

3. Data Ownership and Service Boundaries

A service should own the data whose invariants it is responsible for. Ownership means authority to validate, change, and publish facts—not necessarily exclusive physical possession of every copy. Other services may maintain read models or caches, but they should not directly update the owner’s tables.

For example, an Order service can own order status and line items; a Payment service can own payment authorization; and an Inventory service can own stock reservations. The Order service should not query the Payment service’s database directly to determine whether a payment is valid. It should consume a documented API or event.

Ownership reduces temporal coupling and prevents hidden dependencies. It also requires deliberate duplication. A customer-support read model may contain order and payment summaries, but those values are projections derived from authoritative services. Data duplication is acceptable when provenance, refresh behavior, and conflict handling are understood.

A service boundary should align with business capabilities and transaction boundaries. Splitting a tightly coupled set of tables into separate services without a clear ownership model often creates distributed joins, excessive network calls, and fragile coordination.

4. Replication and Consistency

Replication stores copies of data on multiple nodes or locations. It can improve availability, durability, read performance, and geographic access. It also creates the possibility that different readers see different versions temporarily.

4.1 Consistency models

• Strong consistency: a successful read reflects the latest committed write according to the system’s guarantee.
• Eventual consistency: if writes stop, replicas converge, but a read may temporarily return an older value.
• Read-your-writes consistency: a client sees its own successful updates.
• Monotonic reads: a client does not move backward to an older version after seeing a newer one.
• Causal consistency: causally related operations are observed in an order that respects their dependencies.

The suitable model depends on the business rule. A product description may tolerate eventual consistency; a withdrawal from an account generally requires stronger controls. Consistency is also a per-operation or per-query design choice in many systems, not a single property of the whole application.

4.2 CAP and practical trade-offs

The CAP theorem concerns a distributed data system that experiences a network partition. During a partition, a system cannot simultaneously guarantee both availability for every request and strong consistency for all conflicting operations. It must choose how to respond to the partition. CAP does not mean that a system can have only two of consistency, availability, and partition tolerance in ordinary operation; network partitions are the condition under which the trade-off becomes unavoidable.

Latency and geographic distance add further constraints. A design may choose a quorum, a leader, conflict resolution, or temporary rejection of writes. Engineers should document what happens during partition, failover, and replica lag rather than relying on the label “highly available.”

5. Idempotency and Reliable Retries

Retries are necessary because functions time out, clients disconnect, and messages can be redelivered. A retry is safe when repeating the same logical request has the same intended effect as performing it once. This property is idempotency.

An idempotency key identifies the logical operation, such as payment-attempt-8472. The service stores the key with the result or operation state. On a repeated request, it returns the previously recorded result rather than charging again. The key must be scoped appropriately, protected against misuse, and retained for at least the period in which duplicates may occur.

An idempotency record should be written atomically with the local business change where possible. A common pattern is a database table with a unique constraint on (client_id, idempotency_key). If two requests race, only one can create the record; the other retrieves the existing outcome. External side effects require additional care. For example, a payment provider should receive its own idempotency key, and a message consumer should use a deduplication record or an atomic state transition.

Idempotency is not the same as “the code has no visible duplicate.” A function that increments a counter is not idempotent, because repeating it increments twice. Setting a resource to SHIPPED is usually idempotent if the transition and authorization rules are enforced.

6. Distributed Transactions and the Saga Pattern

A distributed transaction coordinates changes across multiple services. Two-phase commit can provide atomic commitment in some environments, but it introduces a coordinator, holding locks or prepared state, and increased failure and latency concerns. It is often unsuitable for independent cloud services and external providers.

A Saga divides a business transaction into a sequence of local transactions. Each local transaction commits independently and emits an event or response. If a later step fails, compensating actions attempt to undo the business effect of earlier steps.

For a purchase saga:

1. Order service creates an order in PENDING state.
2. Inventory service reserves stock.
3. Payment service authorizes payment.
4. Order service confirms the order.
5. If payment fails, Inventory releases the reservation and Order marks the order CANCELLED.

A compensation is not a perfect rollback. A payment refund may take time, an email cannot be unsent, and a shipment may already be in transit. Therefore, states should be explicit, transitions auditable, and reconciliation possible.

Orchestration and choreography

In orchestration, a coordinator directs each step and records progress. This makes the workflow visible and facilitates timeouts, retries, and compensation, but the coordinator can become complex or overly central.

In choreography, services react to events without a central controller. This can reduce coupling and support independent evolution, but the overall process may become difficult to understand and debug. Event names, ownership, correlation identifiers, schemas, and failure policies are essential.

The transactional outbox pattern helps publish events reliably. A service writes its local business change and an outbox record in one local transaction. A separate publisher reads unsent outbox records and sends them to a broker, marking them published afterward. Because a crash may occur after publishing but before marking, consumers must still be idempotent. The outbox provides at-least-once publication, not magical exactly-once business effects.

7. CQRS and Event Sourcing

Command Query Responsibility Segregation (CQRS) separates operations that change state from operations that read state. A command model validates business decisions and writes authoritative state; one or more query models are optimized for particular views. The read model may be built asynchronously from events, so it can be temporarily stale.

CQRS is valuable when read and write workloads differ substantially, when complex views need specialized projections, or when independent scaling is important. It is not required for every microservice. Unnecessary CQRS adds projection pipelines, rebuild procedures, operational cost, and consistency delay.

Event sourcing stores a sequence of domain events as the source of truth rather than storing only the current state. Current state is reconstructed by replaying events, often with periodic snapshots for efficiency. Events should represent meaningful facts, such as StockReserved or PaymentAuthorized, not low-level implementation details.

Event sourcing supports auditability, temporal analysis, and new projections. Its challenges include event-schema evolution, privacy deletion, storage growth, replay duration, correction of erroneous events, and the distinction between a historical fact and a current business interpretation. Personally identifiable information in immutable events requires a retention and erasure strategy; encryption or tokenization does not automatically satisfy every legal deletion obligation.

8. Applied Case: Illustrative Online Grocery Platform

Background

An online grocery platform uses serverless functions for checkout, a relational database for orders, a key-value store for carts, object storage for product images, a cache for catalogue reads, and a streaming service for domain events.

Problem

Customers sometimes submit checkout twice after a slow network response. Inventory is held in one region, payment is provided by an external gateway, and notification processing occasionally fails. The platform must avoid double charges and must not confirm orders without a valid stock reservation and payment authorization.

Analysis

The Order service owns order status and line items. Inventory owns stock quantities and reservations. Payment owns authorization records. The cart is temporary and can be stored in a key-value service. Product images belong in object storage, while searchable product metadata remains in a database. Catalogue reads can use a cache because a few seconds of staleness is acceptable.

The checkout request carries an idempotency key. The Order service creates one pending order and an outbox event. A Saga coordinator requests inventory reservation, then payment authorization. Each request includes the order identifier as a correlation key and an operation-specific idempotency key.

Approach and solution

The workflow uses these states:

PENDING → STOCK_RESERVED → PAYMENT_AUTHORIZED → CONFIRMED

Failure transitions include:

STOCK_RESERVED → CANCELLING → CANCELLED

and

PAYMENT_AUTHORIZED → REFUND_PENDING → CANCELLED.

Inventory reservations have an expiry time so abandoned checkouts do not hold stock indefinitely. A scheduled function scans for expired reservations, but the expiration operation is idempotent. Payment events are deduplicated using a unique provider transaction identifier. Notifications consume OrderConfirmed events and record a notification key before sending, reducing duplicate messages.

A support read model combines order, payment, and delivery events. It is explicitly labelled as a projection and can be rebuilt. Reconciliation jobs compare provider payment reports with local records and raise exceptions for human review.

Outcome and learning points

The design accepts short periods of eventual consistency while preserving critical invariants: one logical checkout cannot create two orders, one payment attempt cannot be applied twice, and inventory is not confirmed without a reservation. It also recognizes that compensation is a business process rather than a database rollback.

The case demonstrates five transferable lessons: define ownership first; use local transactions within each service; make every retryable boundary idempotent; publish changes reliably through an outbox or equivalent; and operate reconciliation as a normal part of the system rather than as an emergency afterthought.

9. Governance, Security, and Operations

Data architecture must address confidentiality, integrity, availability, retention, and accountability. Apply least-privilege identities to functions, databases, buckets, and streams. Encrypt data in transit and at rest, manage keys securely, and separate development data from production personal data. Audit administrative access and sensitive business operations.

Schema evolution requires compatibility rules. Events should include a version or a documented evolution strategy. Consumers should tolerate added fields and should reject or quarantine malformed messages without blocking an entire stream. Dead-letter handling must include monitoring and a replay procedure.

Operational observability should connect a request identifier, correlation identifier, event identifier, and idempotency key. Useful metrics include consumer lag, retry rate, outbox age, dead-letter count, cache hit rate, replica lag, Saga duration, compensation rate, and reconciliation exceptions. Logs should avoid passwords, payment credentials, and unnecessary personal information.

Cost and sustainability also matter. Retaining every event forever may be unjustified; high-resolution streams, cross-region replication, and excessive cache capacity have financial and environmental costs. Data classification, retention schedules, backup testing, recovery objectives, and residency requirements should be documented before implementation.

10. Best-Practice Design Method

A practical selection process is:

1. Identify the business capability and authoritative owner.
2. List invariants, transaction boundaries, and acceptable stale-read periods.
3. Document read, write, volume, latency, retention, and query patterns.
4. Choose the simplest data model that satisfies those requirements.
5. Define replication, failure, retry, and recovery behavior.
6. Add idempotency at public commands and message consumers.
7. Use an outbox or equivalent when a local change must produce an event.
8. Select Saga orchestration or choreography deliberately.
9. Design projections, reconciliation, and observability before production.
10. Test duplicate delivery, timeouts, partial failure, replica lag, partition, and replay.

Technology selection should be revisited as access patterns and scale change. A managed service reduces infrastructure work, but it does not remove responsibility for data modelling, security, correctness, or operations.

Challenges and Limitations

Distributed data systems increase autonomy and scale but impose cognitive and operational cost. Eventual consistency can confuse users unless interfaces communicate processing status. Sagas cannot guarantee instantaneous atomicity across independent providers. CQRS and event sourcing add moving parts and may be inappropriate for simple CRUD applications. Caches create invalidation risk, replicas create lag, and streams require schema and replay discipline. Finally, “exactly once” claims should be examined carefully: transport delivery, processing, and business effects are different guarantees.

Chapter Summary

Cloud-native applications use a portfolio of data services. Relational systems provide strong structure and transactions; NoSQL systems optimize particular access patterns; object storage handles large files; caches reduce latency; and streams support durable asynchronous integration. Each service should have a clear data owner, while derived copies must have documented provenance and freshness.

Replication improves resilience but introduces consistency choices. Distributed workflows should use local transactions, idempotent commands and consumers, correlation identifiers, and reliable event publication. Sagas coordinate local transactions and compensations, while CQRS separates write decisions from read projections. Event sourcing records domain facts and enables replay, but requires careful evolution, privacy, and operational practices.

The professional goal is not maximum technological sophistication. It is a system whose invariants, failure behavior, governance, and operational costs are explicit and appropriate to the business.

Key Terms

Data ownership: Authority and responsibility for validating and changing a data domain.

ACID: Atomicity, consistency, isolation, and durability properties of a transaction.

Relational database: A structured, table-oriented data system supporting relationships and transactions.

NoSQL: A group of non-relational data models optimized for particular access patterns.

Object storage: Durable storage for named binary or file-like objects.

Cache: A temporary, faster copy used to reduce latency or source-system load.

Stream: A retained, ordered sequence of records processed by consumers.

Replication: Maintaining copies of data on multiple nodes or locations.

Eventual consistency: A model in which replicas converge after updates stop.

Idempotency: The property that repeating a logical operation has the same intended effect as doing it once.

Saga: A sequence of local transactions coordinated with compensating actions.

Compensation: A business operation that counteracts an earlier completed action.

Orchestration: Saga coordination by a central workflow component.

Choreography: Coordination through decentralized event reactions.

Transactional outbox: A local record written with a business change and later published as an event.

CQRS: Separation of command/write models from query/read models.

Event sourcing: Storing domain events as the authoritative history of state changes.

Projection: A derived read model built from authoritative data or events.

Correlation identifier: An identifier used to connect related requests, events, and logs.

Reconciliation: Comparing independent records to detect and resolve discrepancies.

Review Questions

Short Answer Questions

1. Why should a microservice generally avoid directly updating another service’s database?
2. Distinguish a cache from an authoritative data store.
3. What problem does an idempotency key solve?
4. Define eventual consistency and give one suitable use case.
5. What is the purpose of a transactional outbox?

Long Answer Questions

1. Compare relational, document, key-value, object, cache, and streaming services for cloud application design.
2. Explain how data ownership and service boundaries affect consistency and coupling.
3. Describe a Saga workflow, including orchestration, choreography, and compensation.
4. Explain CQRS and event sourcing, including their benefits and limitations.
5. Discuss the operational and governance requirements of a distributed data architecture.

Analytical / Application Questions

1. Design a data architecture for a ticket-booking platform. Identify owners, stores, consistency requirements, and a strategy for preventing double booking.
2. A message consumer sometimes applies the same PaymentAuthorized event twice. Propose an idempotent processing design and explain its failure cases.
3. An organization wants to introduce event sourcing for a simple employee directory. Evaluate whether the choice is justified, and recommend an alternative if it is not.

Multiple Choice Questions with Answers

Question 1

Which data service is generally most appropriate for enforcing foreign-key relationships and multi-record transactions?

A. Relational database

B. Object storage

C. Distributed cache

D. Image CDN

Correct Answer: A. Relational database

Question 2

What is the primary purpose of an idempotency key in a payment request?

A. To encrypt the payment amount

B. To prevent a repeated logical request from creating a duplicate effect

C. To increase database storage capacity

D. To guarantee that a network never fails

Correct Answer: B. To prevent a repeated logical request from creating a duplicate effect

Question 3

In a Saga, what normally happens when a later local transaction fails?

A. All databases are automatically rolled back by one global transaction

B. The system deletes every historical event

C. Compensating actions are attempted for earlier completed steps

D. The service ignores the failure

Correct Answer: C. Compensating actions are attempted for earlier completed steps

Question 4

Which statement best describes a transactional outbox?

A. It stores cache entries only in browser memory

B. It writes a business change and an event record in the same local transaction

C. It guarantees exactly-once delivery to every consumer

D. It replaces all database backups

Correct Answer: B. It writes a business change and an event record in the same local transaction

Question 5

A product catalogue may commonly use eventual consistency because:

A. Every catalogue read must be globally serialized

B. Short-lived stale descriptions are usually less harmful than high coordination latency

C. Eventual consistency prevents all duplicate writes

D. It removes the need for authorization

Correct Answer: B. Short-lived stale descriptions are usually less harmful than high coordination latency

Question 6

In CQRS, a query model is primarily designed to:

A. Optimize reading particular views of data

B. Authorize payments at a bank

C. Replace all command validation

D. Store encryption keys in source code

Correct Answer: A. Optimize reading particular views of data

Question 7

Which storage service is best suited to large product images and downloadable documents?

A. Object storage

B. A relational join table only

C. A message consumer offset

D. A DNS cache

Correct Answer: A. Object storage

Question 8

Why can a stream consumer still need deduplication when the stream is durable?

A. Durability means messages are never stored

B. A crash can occur after processing but before the consumer records its progress

C. Streams cannot preserve any ordering

D. Deduplication is required only for images

Correct Answer: B. A crash can occur after processing but before the consumer records its progress

Question 9

Which design most directly prevents two checkout submissions with the same logical operation from creating two orders?

A. A unique idempotency key enforced with an atomic record

B. Increasing image resolution

C. Disabling all read replicas

D. Using a longer product description

Correct Answer: A. A unique idempotency key enforced with an atomic record

Question 10

What is a key limitation of compensation in a distributed Saga?

A. It always restores the exact physical state instantly

B. It is a business action and may be delayed or impossible to reverse perfectly

C. It can operate only inside object storage

D. It eliminates the need for monitoring

Correct Answer: B. It is a business action and may be delayed or impossible to reverse perfectly