Modern software rarely operates as an isolated program. Mobile applications call web services, payment systems communicate with banks, warehouses exchange data with marketplaces, and analytical platforms consume streams of operational events. These interactions are enabled by application programming interfaces (APIs), integration patterns, messaging systems, and distributed-system protocols.

Distributed systems create capabilities that a single process cannot easily provide: horizontal scalability, geographic availability, independent deployment, and integration across organizational boundaries. They also introduce difficult problems. Communication is slower and less reliable than a local function call; messages may be duplicated or delayed; clocks disagree; services may fail independently; and a successful operation in one component may coincide with failure in another.

This chapter develops the foundations needed to design robust REST, RPC, GraphQL, messaging, and event-streaming solutions. It also examines latency, partial failure, time, idempotency, consistency, transactions, consensus, and CAP-related trade-offs.

Learning Objectives

1. APIs as Contracts Between Systems

An API is a defined contract through which one software component requests data or behavior from another. A useful contract specifies:

An API is more than an endpoint or method signature. It is a social and technical agreement between producers and consumers. Changing a field from optional to mandatory, altering the meaning of a status value, or removing an error code can break clients even when the server still compiles.

Resource-oriented APIs and REST

REST is an architectural style based on resources, representations, uniform interfaces, stateless requests, and the separation of client and server responsibilities. HTTP commonly maps operations to methods:

HTTP method	Typical meaning	Idempotency expectation
GET	Retrieve a representation	Yes
POST	Create or invoke a non-idempotent operation	Usually no
PUT	Replace a resource at a known URI	Yes
PATCH	Partially modify a resource	Depends on design
DELETE	Remove a resource	Yes

An operation is idempotent when repeating it has the same intended effect as performing it once. This does not mean every response is identical. For example, the first DELETE may return 204 while a repeated DELETE may return 404, yet the resource remains absent.

A good REST design uses nouns for resources, meaningful HTTP status codes, pagination for collections, and explicit representations. For example, GET /customers/42/orders?limit=20&cursor=... is usually clearer than /getCustomerOrders. Pagination should use stable ordering and an opaque cursor when records can change during traversal.

Conditional requests reduce lost updates. A server may return an entity tag (ETag), and a client may submit If-Match: "v17". The server performs the update only if the representation is still version 17. This is optimistic concurrency control.

RPC

Remote procedure call (RPC) presents communication as invoking a function on another machine. It is often effective for internal service-to-service operations where a precise schema, generated client, and low serialization overhead are valuable. Protocol Buffers with gRPC, for example, support strongly typed contracts and streaming.

RPC can mislead developers into treating a network operation like a local call. A remote call has serialization cost, network latency, authentication overhead, timeouts, and partial-failure behavior. Therefore, interfaces should expose coarse-grained operations and explicit deadlines rather than requiring many fine-grained calls.

GraphQL

GraphQL allows a client to specify the fields it needs through a typed schema. It can reduce over-fetching and under-fetching, particularly for user interfaces aggregating several domain objects. However, flexible queries create governance challenges: expensive nested queries can overload servers, authorization must be enforced at field and resolver boundaries, and caching is less straightforward than caching a REST resource.

Production GraphQL services commonly apply query depth or cost limits, persisted queries, pagination, batching, and resolver instrumentation. GraphQL is an application query interface; it does not eliminate the need for sound domain boundaries or reliable back-end services.

2. API Reliability and Lifecycle Design

Errors and retries

Errors should distinguish client faults, authentication failures, conflicts, rate limits, and server failures. A useful error response includes a stable machine-readable code, a human-readable message, a correlation identifier, and, where appropriate, retry guidance. Sensitive implementation details and stack traces should not be exposed.

Retries are appropriate only when the failure is plausibly transient and the operation is safe to repeat. Exponential backoff with jitter prevents many clients from retrying simultaneously. A common schedule is:

Here, D_k is the delay before retry k, D₀ is the initial delay, D_max is the maximum delay, and J is a random jitter value. The formula assumes that retries are bounded and that the service may recover; it is not a substitute for load shedding or circuit breaking.

A timeout bounds how long a caller waits. Deadlines are preferable to independent timeouts because a request’s remaining budget can be propagated through downstream calls. If a request has deadline T_d and the current time is T_now, its remaining budget is:

The quantities must use the same clock and time unit. A downstream call must not receive a longer deadline than the caller’s remaining budget.

Latency and throughput

For a sequential path, approximate end-to-end latency is the sum of component latencies:

This approximation assumes the stages are sequential and excludes queueing and variability. Parallel calls behave differently: the critical-path latency is approximately the maximum of the parallel branches, plus coordination overhead:

Tail latency matters because a system serving many requests is affected by slow samples. If five independent downstream calls each succeed within a target time with probability 0.99, and all must finish before the request succeeds, the idealized combined probability is:

This assumes independence, identical probabilities, and no retries or shared bottlenecks. Real systems often violate these assumptions, so measured traces are essential.

Throughput and latency are related but not interchangeable. Queueing increases latency as utilization approaches capacity. Engineers should measure request rate, saturation, error rate, and latency percentiles such as p50, p95, and p99 rather than relying only on averages.

Bulkheads, circuit breakers, and rate limits

A bulkhead isolates resources so that failure in one workload does not exhaust all threads, connections, or memory. A circuit breaker temporarily stops calls to an unhealthy dependency, allowing recovery and protecting callers. Rate limiting controls demand, often using token-bucket or leaky-bucket mechanisms. These controls should produce explicit, observable responses such as HTTP 429 rather than silently discarding work.

3. Messaging and Event Streaming

A synchronous request requires the caller to wait for a response. Messaging decouples sender and receiver through a broker or queue. A message usually contains a payload, metadata, a schema identifier, a message identifier, a timestamp, and a causation or correlation identifier.

Queues and publish-subscribe

In a work queue, one message is normally processed by one consumer in a consumer group. In publish-subscribe, multiple subscribers receive their own logical copy. A queue is appropriate for distributing tasks; a topic or stream is appropriate when several independent consumers need the same business event.

Event notification should describe a fact that occurred, such as OrderPaid. A command requests an action, such as AuthorizePayment. Confusing commands with events creates unclear ownership and makes replay difficult.

Delivery semantics

Model	Meaning	Engineering consequence
At-most-once	A message is delivered zero or one time	Possible loss; minimal duplication
At-least-once	A message is retried until acknowledged	Duplicates are possible
Effectively-once	Duplicate effects are prevented by application design	Requires idempotency or deduplication
Exactly-once processing	One logical processing result under a precisely defined system scope	Strong assumptions and limited scope

“Exactly once” must always be qualified. A broker may avoid duplicate records internally, while an external email provider or payment gateway can still observe repeated requests. At-least-once delivery combined with idempotent consumers is usually more practical.

An idempotent consumer records a business key or event identifier before, during, or atomically with applying the effect. For example, a payment service can enforce a unique constraint on payment_request_id. A retry then returns the original result rather than charging again.

Event streams

An event stream is an ordered, append-oriented sequence of records. Partitioning enables parallelism: records with the same key are routed to the same partition, preserving order for that key. Global ordering across all partitions is expensive and often unnecessary.

Consumers track offsets or checkpoints. Replaying from an earlier offset supports recovery, debugging, and rebuilding derived views. This makes schema evolution, retention policy, privacy deletion, and deterministic processing important design concerns.

The transactional outbox pattern addresses a common failure: a service commits a database change but crashes before publishing the corresponding event. The service writes the business change and an outbox record in one local database transaction. A relay later publishes the outbox record and marks it dispatched. Consumers remain idempotent because relay retries can publish duplicates.

4. Time, Ordering, and Partial Failure

Distributed systems cannot assume a single perfectly synchronized clock. Wall-clock time can jump because of synchronization or administrator changes. Monotonic clocks are appropriate for measuring durations within a process. Timestamps are useful for display and approximate ordering but are not proof of causality.

A happened-before relation captures causal order: an event precedes another if it occurs earlier in the same process, a message is sent before it is received, or order is transitive. Concurrent events may have no causal order. Logical clocks and version vectors can represent aspects of this relationship without claiming that physical timestamps are exact.

Partial failure is a defining distributed-systems problem. One service may be healthy while a network path is broken; a response may be lost after the server committed the operation; or a dependency may be slow rather than unavailable. A timeout does not prove that the operation failed—it proves only that the caller lacks confirmation.

This distinction explains why retrying a POST can create duplicates. Safe retry requires one of the following: an idempotent operation, an idempotency key whose result is stored, a transactional protocol, or a compensating action.

5. Consistency, Replication, and Transactions

Consistency describes what reads may observe after writes. Strong consistency aims for behavior equivalent to a single current copy. Eventual consistency allows replicas to converge if updates stop, but reads may temporarily differ. Causal consistency preserves causally related order while allowing some concurrent operations to appear in different orders.

Replication improves availability and read capacity but introduces lag and conflict. A primary-replica design may route writes to one primary and reads to replicas. Read-after-write requirements may require reading from the primary, waiting for replication, or attaching a session version to the request.

Transactions

A local database transaction commonly provides atomicity, consistency according to database constraints, isolation, and durability (ACID). Distributed transactions are harder because participants can fail independently.

Two-phase commit (2PC) has a prepare phase and a commit phase. A coordinator asks participants to prepare; if all agree, it announces commit, otherwise abort. 2PC can preserve atomicity but may block if the coordinator fails, and it imposes coordination and availability costs.

Sagas decompose a business transaction into local transactions connected by messages. Each step has a compensating action. Compensation is not necessarily a perfect inverse: a shipment cancellation may require a refund and customer notification rather than restoring an earlier physical state. Sagas are often preferable for long-running workflows, but they require explicit state machines, retries, reconciliation, and attention to intermediate visibility.

6. Consensus and CAP-Related Trade-offs

Consensus is agreement among distributed participants despite failures, usually concerning an ordered log or a selected value. Practical consensus protocols such as Raft use leader election, terms, replicated logs, and quorum acknowledgments. Consensus is not the same as replication: replicas may copy data asynchronously without agreeing on a single authoritative order.

CAP reasoning concerns a distributed data system facing a network partition. The three properties are:

When a partition occurs, a system cannot simultaneously guarantee strong consistency and availability for all operations. Since real networks can partition, the practical decision is usually how to behave during partitions: reject or delay some operations to preserve consistency, or accept operations and reconcile later to preserve availability. CAP does not state that a system has only two permanent properties, nor does it replace analysis of latency, durability, semantics, or operational needs.

For example, an inventory reservation may prefer consistency to avoid overselling, while a social “like” counter may prefer availability and reconcile approximate totals. Different data within the same product may use different policies.

7. Security, Governance, and Observability

API security begins with authentication, but authentication alone does not establish authorization. Authorization should verify the caller’s permitted action on the specific resource, using least privilege. OAuth 2.0 and OpenID Connect are common foundations, but correct token validation, audience checking, expiry handling, and key rotation remain the implementer’s responsibility.

Transport encryption, secret management, input validation, output encoding, and protection against replay are essential. Sensitive data should be minimized in payloads and logs. Personal data requires retention, deletion, access control, and lawful processing consistent with applicable regulation.

Contracts should be documented using machine-readable specifications where practical. Consumer-driven contract tests can detect incompatible changes. Backward-compatible evolution generally adds optional fields, preserves existing meanings, and avoids reusing removed identifiers. Versioning should be intentional; URL, header, or media-type versioning each has operational consequences.

Observability combines logs, metrics, and traces. Correlation IDs connect a user request to downstream calls and messages. Metrics should include traffic, errors, latency, saturation, queue depth, consumer lag, retry counts, and dead-letter volume. Distributed tracing should preserve causal context without placing secrets or excessive personal data into trace attributes.

8. Applied Case: An Illustrative Online Retail Order Platform

Background and problem

Consider a fictional retailer whose web application calls order, payment, inventory, shipping, and notification services. The original design performs five synchronous calls during checkout. Customers sometimes receive timeouts even though payments have succeeded, and inventory becomes inconsistent during service outages.

Analysis

The design has a long sequential critical path, so latency is approximately the sum of gateway, service, network, and database times. A timeout after payment authorization creates ambiguity: the client cannot know whether the payment was committed. Retrying without protection could charge the customer twice. A single distributed transaction across payment, inventory, and shipping would be operationally expensive and could block during failures.

Approach

The client receives a durable order identifier and an explicit status such as PENDING, rather than waiting for every downstream action. A later query or notification reports completion. The payment provider is called with the same idempotency key on retries.

Outcome and learning points

The architecture reduces synchronous coupling and prevents common duplicate effects, but it does not make the workflow instantaneous or magically atomic. Customers may temporarily see a pending order. Reconciliation remains necessary because external providers and networks can fail in ambiguous ways. The case demonstrates that reliability is achieved through contracts, durable state, idempotency, workflow design, observability, and operational recovery—not through a single protocol.

9. Best Practices and Limitations

No pattern is universally superior. REST may be ideal for public resource APIs, RPC for controlled internal calls, GraphQL for client-shaped queries, queues for work distribution, and streams for durable event histories. Combining them is normal, but each boundary increases operational and governance complexity.

Contemporary systems increasingly use schema registries, service meshes, workflow engines, and policy-as-code. AI services add further integration concerns: nondeterministic outputs, prompt and data privacy, model latency, provider outages, evaluation drift, and the need to treat generated content as untrusted input. These concerns should be addressed with ordinary engineering controls—timeouts, validation, authorization, auditability, and human oversight—rather than assuming that an AI component is reliable by default.

Chapter Summary

APIs are contracts governing data, behavior, errors, security, compatibility, and operational expectations. REST models resources and uses uniform HTTP semantics; RPC provides typed remote operations; GraphQL lets clients select fields but requires query governance. Messaging decouples producers and consumers, while event streams provide durable, replayable histories.

Distributed calls differ from local calls because latency, timeouts, duplication, reordering, and partial failure are unavoidable possibilities. Deadlines, backoff, circuit breakers, bulkheads, and rate limits control these risks. At-least-once delivery makes idempotent consumers and deduplication essential. Outbox patterns connect local database state to events, while sagas coordinate long-running business workflows through compensations.

Replication creates consistency choices. Two-phase commit provides coordinated atomicity at significant availability and operational cost; consensus establishes agreement and ordered state but is not synonymous with replication. CAP explains the unavoidable choice between strong consistency and availability during a partition. Effective engineering therefore begins with business invariants, failure behavior, observability, security, and governance rather than with protocol popularity.

Key Terms

REST: An architectural style using resources, representations, and uniform interfaces.

GraphQL: A typed query language and execution system in which clients select requested fields.

Idempotency: The property that repeating an operation has the same intended effect as performing it once.

Partial failure: Failure of some distributed components while others continue operating.

At-least-once delivery: Delivery in which messages may be delivered repeatedly but are retried after failure.

Consumer group: A set of consumers sharing work for a queue or partitioned stream.

Outbox pattern: Storing a domain change and its pending event in one local transaction.

CAP: A model of trade-offs among consistency, availability, and partition tolerance during partition.

Circuit breaker: A mechanism that temporarily stops calls to an unhealthy dependency.

Consumer lag: The distance between the latest stream record and a consumer’s processed position.

Review Questions

Short Answer Questions

Long Answer Questions

Analytical / Application Questions

Multiple Choice Questions with Answers

Question 1

Which HTTP method is normally used to retrieve a resource without changing server state?

Question 2

Correct Answer: B. To prevent repeated submissions from creating duplicate effects

Question 3

Which delivery model permits duplicate message delivery and therefore commonly requires idempotent consumers?

Question 4

What does the transactional outbox pattern place in the same local transaction as a business change?

Question 5

For five independent downstream calls, each succeeding within a target with probability 0.99, what is the idealized probability that all succeed within the target?

Question 6

Correct Answer: B. A sequence of local transactions with compensating actions

Question 7

During a network partition, CAP reasoning identifies a trade-off primarily between which two guarantees?

Question 8

B. It can preserve ordering for records sharing that key while enabling parallelism

Correct Answer: B. It can preserve ordering for records sharing that key while enabling parallelism

Question 9

What does a circuit breaker normally do when a dependency is repeatedly failing?

Question 10

B. Consensus is agreement on a value or order, whereas replication maintains copies

Correct Answer: B. Consensus is agreement on a value or order, whereas replication maintains copies

Chapter 5 APIs, Integration and Distributed Systems

APIs, Integration and Distributed Systems

Introduction