APIs, Integration and Distributed Systems

Introduction and Context

Modern full-stack applications rarely operate as isolated programs. A browser application may call an identity provider, payment service, search engine, notification platform, analytics system, and several internal services. These interactions are enabled by application programming interfaces (APIs) and integration mechanisms. Distributed systems extend this problem: components execute on different machines, communicate over networks, fail independently, and may observe different versions of data.

Effective integration is therefore more than sending an HTTP request. Engineers must define contracts, select suitable interaction styles, manage change, handle retries safely, preserve security and privacy, and reason about consistency and failure. This chapter examines REST, GraphQL, remote procedure call (RPC), webhooks, message queues, and streaming systems as complementary design choices.

Learning Objectives

By the end of this chapter, students should be able to:

1. Compare REST, GraphQL, RPC, webhooks, messaging, and streaming.
2. Design API contracts using schemas, validation, status semantics, and documentation.
3. Explain versioning and schema compatibility accurately.
4. Apply idempotency, retries, timeouts, circuit breakers, and dead-letter handling.
5. Distinguish consistency models and select appropriate integration patterns.
6. Evaluate a distributed integration for performance, security, observability, and governance.

1. Distributed Systems as Integration Environments

A distributed system consists of components that communicate through a network and do not share one global memory or clock. Network latency, packet loss, duplicated messages, partial outages, clock skew, and independent deployment create uncertainty absent from a single process.

A useful design principle is to assume that every remote operation can:

• succeed;
• fail before the caller receives a response;
• be processed but have its response lost;
• be delivered more than once; or
• become unavailable for an unpredictable period.

This is why a client timeout does not prove that a server did not perform an operation. The client knows only that it did not receive a response within the selected time.

Integration dimensions

Integration choices can be evaluated across several dimensions:

2. API Contracts

An API contract specifies how independent software components interact. It commonly includes endpoints or methods, parameters, representations, schemas, authentication, error formats, rate limits, and behavioral guarantees.

A strong contract is:

• explicit rather than dependent on undocumented behavior;
• machine-readable where practical;
• testable through contract tests;
• stable enough for independent release; and
• precise about failure and retry semantics.

For HTTP APIs, an OpenAPI document may describe paths, operations, parameters, request bodies, responses, and security schemes. JSON Schema can describe data constraints such as required properties, types, ranges, and patterns. A schema, however, does not fully specify behavioral properties such as whether a POST is idempotent or whether an operation is eventually consistent. Those guarantees must also be documented.

Representation and semantic status

HTTP methods communicate intent. GET retrieves a representation, PUT replaces a representation at a known URI, PATCH applies a partial modification, and DELETE requests removal. POST commonly creates a subordinate resource or initiates an operation. Idempotency is a behavioral property: repeating an operation has the same intended effect as performing it once. GET, PUT, and DELETE are conventionally idempotent when correctly implemented; POST is not inherently idempotent.

A useful error representation should distinguish validation failure, authentication failure, authorization failure, missing resources, conflict, rate limiting, and server failure. Returning HTTP 200 for every outcome makes automated handling and observability more difficult.

3. Integration Styles

3.1 REST

Representational State Transfer (REST) uses resources, representations, uniform interfaces, and stateless requests. A REST client requests a representation of a resource or submits a state transition. HTTP caching, content negotiation, conditional requests, and standard status codes can be used naturally.

REST is suitable for public APIs, CRUD-oriented domains, and integrations where broad tooling and cacheability matter. Its limitations include over-fetching or under-fetching, multiple round trips for related data, and ambiguity when a domain operation is not naturally a resource update.

3.2 GraphQL

GraphQL exposes a typed schema and allows a client to request a shaped response. A query can retrieve related fields in one operation, while mutations represent changes and subscriptions can provide selected updates.

GraphQL can reduce client-specific endpoint proliferation, but it introduces resolver complexity, authorization challenges at field level, and risks such as expensive or deeply nested queries. Query depth, cost limits, pagination, batching, and persisted queries are important controls. GraphQL does not automatically solve consistency, caching, or authorization.

3.3 RPC

RPC models communication as calling a remote procedure, often with generated clients and strongly typed messages. gRPC, for example, commonly uses Protocol Buffers over HTTP/2 and supports unary and streaming calls.

RPC is effective for internal service-to-service calls requiring efficient serialization, explicit interfaces, and generated code. It can encourage the false belief that a remote call behaves like a local function. Timeouts, retry policy, cancellation, authentication, and partial failure must remain explicit.

3.4 Webhooks

A webhook is an outbound HTTP callback triggered by an event. Instead of repeatedly polling a provider, a subscriber exposes an endpoint and receives notifications such as payment.completed.

A robust webhook receiver verifies a signature, checks timestamp or replay protection, validates the schema, records an event identifier, performs quick acknowledgement, and processes work asynchronously. Providers may retry delivery, so the receiver must tolerate duplicates. The receiver should return a success response only when it has durably accepted the event, not merely parsed it in memory.

3.5 Messaging and queues

A message broker decouples producers and consumers. A queue generally distributes work among competing consumers; a topic or log allows multiple consumer groups to receive the same event independently. Messages may contain commands (“reserve stock”) or events (“stock reserved”). Commands express desired action; events report a fact that has occurred.

At-least-once delivery is common and requires idempotent consumers. A dead-letter queue stores messages that repeatedly fail, allowing investigation without blocking healthy traffic. Ordering is usually guaranteed only within a defined scope, such as a partition or key, not globally.

3.6 Streaming

Streaming systems process an ordered, durable sequence of records, often partitioned for scale. Consumers track offsets and may replay historical data. Streaming is useful for analytics, audit trails, materialized views, and high-volume event processing.

A stream is not automatically a database of truth. Retention, compaction, event ordering, late arrivals, schema evolution, privacy deletion, and replay behavior must be designed. Event time and processing time may differ; windowed calculations need explicit handling of late records.

Figure: Synchronous request–response integration compared with asynchronous broker-based distribution.

4. Versioning and Compatibility

Change management begins with distinguishing additive, modifying, and destructive changes. Adding an optional response field is often safe; renaming a field or changing its type is usually breaking. Removing an endpoint, changing error semantics, or narrowing accepted values can also break clients.

API versioning strategies include:

• URI versioning, such as /v2/orders;
• media-type or header versioning;
• query-parameter versioning; and
• evolution without a visible version through backward-compatible additions and a deprecation policy.

No strategy eliminates the need to support consumers during migration. Every version should have ownership, documentation, telemetry, a sunset date, and a communication process.

Schema-registry terminology

Compatibility terminology depends on the direction of change. In common schema-registry usage:

• Backward compatibility means a new schema can read data written with the older schema. This protects new consumers or upgraded readers from historical messages.
• Forward compatibility means the older schema can read data written with the newer schema. This supports producers being upgraded before consumers.
• Full compatibility means both directions are supported.

For example, adding a field with a default may support both directions, while removing a field may prevent an old consumer from interpreting newer messages. Exact rules depend on serialization technology and configuration; terminology should be confirmed against the registry’s documented policy.

A practical evolution pattern is to add optional fields, preserve meanings, avoid reusing field identifiers, tolerate unknown fields, and use explicit deprecation periods. Consumer-driven contract tests can verify that a provider change remains acceptable to real consumers.

5. Idempotency, Retries, and Resilience

Idempotency keys

For a potentially repeated command such as charging a card, the client can send an idempotency key. The server stores the key, request fingerprint, resulting status, and response. A repeated request with the same key and equivalent parameters returns the stored result; the same key with different parameters should be rejected. Storage retention must cover the likely retry and reconciliation period.

Idempotency does not mean that all side effects are automatically safe. A payment service may deduplicate the charge, while an email provider may still send duplicates unless it offers a corresponding guarantee.

Timeouts and exponential backoff

Timeouts should be bounded and selected from a latency budget. A retry must be limited by an overall deadline, not merely by a retry count. Exponential backoff reduces synchronized retry storms. Let b be the initial delay, n the zero-based retry number, and C the maximum permitted delay. A capped exponential backoff with uniform jitter J_n, where 0 ≤ J_n ≤ j, is:

Here D_n, b, C, and J_n are durations, and j is also a duration. The cap is applied after adding jitter, so D_n ≤ C. This differs from adding jitter after a capped deterministic delay, which could exceed the stated maximum. The formula assumes a non-negative base and cap with b ≤ C; implementations may use full jitter or another documented policy.

Worked example. Let b = 200 ms, C = 2,000 ms, and choose J_0 = 50 ms, J_1 = 100 ms, J_2 = 150 ms. The delays are:

• D_0 = min(2,000, 200 + 50) = 250 ms;
• D_1 = min(2,000, 400 + 100) = 500 ms;
• D_2 = min(2,000, 800 + 150) = 950 ms.

If the deterministic value plus jitter were 2,100 ms, the result would be min(2,000, 2,100) = 2,000 ms, never 2,100 ms.

Retries are appropriate only for transient failures and operations whose duplication is safe or controlled. Retrying a validation error, authentication failure, or non-idempotent command can worsen the situation. A circuit breaker temporarily stops calls after a failure threshold, allowing an unhealthy dependency to recover. Bulkheads isolate resource pools so one dependency cannot exhaust all threads or connections.

6. Consistency and Failure Models

Strong consistency makes a completed write immediately visible to subsequent reads under the relevant system model. Eventual consistency permits temporary divergence but requires that replicas converge if updates stop. Causal consistency preserves cause-and-effect ordering, while read-your-writes consistency ensures a user can observe their own successful update.

The choice is contextual. A bank ledger may require strict transactional controls, whereas a product recommendation view can tolerate delay. Distributed transactions such as two-phase commit can provide coordination but add blocking, latency, and operational complexity. The saga pattern instead decomposes a business transaction into local transactions with compensating actions. A compensation is a new business operation, not a literal rollback: cancelling a shipment does not erase the fact that it was once dispatched.

The outbox pattern improves reliable publication. A service writes its business change and an outbox record in one local database transaction. A relay later publishes the outbox record. Consumers remain idempotent because relay crashes can cause publication more than once. This pattern reduces, but does not magically eliminate, duplication and reconciliation concerns.

7. Security, Governance, and Observability

API security should apply authentication and authorization separately. OAuth 2.0 and OpenID Connect are commonly used for delegated access and identity, but possession of a valid token does not imply permission for every resource. Use least privilege, audience and issuer validation, short-lived credentials where appropriate, secret rotation, and explicit tenant isolation.

Transport encryption protects data in transit; encryption at rest and carefully managed keys protect stored data. Logs should avoid passwords, access tokens, payment data, and unnecessary personal information. Webhook signatures must be verified against the raw request body, with replay protection and constant-time comparison where applicable.

Governance includes ownership, data classification, retention, consent, deletion, auditability, rate limits, and third-party risk assessment. Privacy regulations may require purpose limitation, access rights, deletion, and breach procedures. Event replay can unintentionally reproduce personal data, so retention and redaction policies must cover brokers and backups.

Observability should correlate traces across services using a trace or correlation identifier. Useful measures include latency percentiles, error rate, saturation, queue age, consumer lag, retry count, dead-letter volume, and schema-validation failures. Averages alone can hide tail latency; the 95th or 99th percentile is often more operationally informative.

8. Selection Framework and Best Practices

Choose REST when resource semantics, broad interoperability, and HTTP tooling dominate. Choose GraphQL when clients need varied projections across related data and the organization can govern query cost. Choose RPC for strongly typed internal calls and streaming procedures. Choose webhooks for provider-to-consumer notifications. Choose queues for work distribution and isolation. Choose durable streams for replayable, high-volume event histories.

A disciplined design process is:

1. Identify whether the interaction is a query, command, event, or continuous measurement.
2. Define ownership and the source of truth.
3. Specify schema, semantics, authentication, errors, and delivery guarantees.
4. State timeout, retry, idempotency, ordering, and consistency behavior.
5. Design versioning, deprecation, and compatibility tests.
6. Model overload, dependency failure, duplication, and recovery.
7. Add metrics, traces, audit records, and alerts.
8. Test contract evolution, replay, chaos scenarios, and security boundaries.

Applied Case: Illustrative Retail Order Integration

Background

An illustrative online retailer has a web storefront, inventory service, payment provider, warehouse system, and customer-notification service. The original design synchronously called all four dependencies during checkout.

Problem

Payment timeouts caused users to retry. Some payments succeeded even though the browser displayed an error. Inventory latency also made checkout slow, and a warehouse outage prevented orders from being created.

Analysis

The design coupled order creation to every downstream system and had no idempotency key. It treated a lost response as a failed payment and had no reconciliation process. Notifications were part of the critical path even though they were not required to establish the order.

Approach

The redesigned flow assigns a client-generated order-attempt key. The order service stores the request and creates an order in a local transaction. It writes OrderCreated and PaymentRequested records to an outbox. A payment adapter consumes the request with the same idempotency key, while inventory reservation is handled by a saga. Warehouse and notification consumers process events asynchronously. Failed messages are retried with bounded backoff and moved to a dead-letter queue after the policy limit. A reconciliation job compares payment-provider records with local order state.

The API returns a state such as PENDING_PAYMENT rather than pretending that all work is complete. Clients poll or receive a webhook-backed status update. Contract tests ensure that optional event fields can be added without breaking existing consumers.

Outcome and learning points

The design reduces synchronous coupling, makes duplicate payment attempts safer, and exposes uncertainty honestly. It does not guarantee instantaneous consistency: the order may briefly exist before inventory or payment status is known. That trade-off is explicit and observable. The central lesson is that resilience comes from combining contracts, idempotency, asynchronous boundaries, reconciliation, and operational evidence—not from retries alone.

Challenges and Limitations

No integration style is universally superior. REST can become fragmented, GraphQL can create expensive resolver workloads, RPC can hide network costs, webhooks depend on publicly reachable receivers, queues can create invisible backlog, and streams require careful partition and replay design. Exactly-once processing is generally an end-to-end business property rather than a simple broker switch. It normally requires durable identities, transactional boundaries, deduplication, and clear business semantics.

Testing distributed systems also has limits. Unit tests cannot reveal all timing and failure interactions. Contract, integration, load, security, fault-injection, and recovery tests should complement them. Teams must balance reliability investment against business impact, latency requirements, and operational capacity.

Contemporary Developments

API gateways increasingly provide authentication, rate limiting, routing, and policy enforcement, while service meshes provide traffic management and telemetry between services. Event-driven architectures are being combined with serverless consumers and managed streaming platforms. Schema registries, data contracts, and data products improve organizational interoperability.

AI services add integration concerns including prompt and model versioning, token limits, nondeterministic outputs, confidential-data handling, cost controls, and evaluation. An AI response should not be treated as a reliable contractual fact without validation and appropriate human or rule-based controls.

Chapter Summary

APIs and distributed integrations connect independently deployed software but introduce latency, partial failure, duplication, version skew, and consistency trade-offs. REST, GraphQL, RPC, webhooks, messaging, and streaming address different interaction needs. Contracts must describe both data and behavior. Safe evolution depends on explicit compatibility rules: backward compatibility commonly means new readers can read old data, while forward compatibility means old readers can read new data. Idempotency keys, bounded timeouts, correctly capped exponential backoff, circuit breakers, bulkheads, outbox records, sagas, dead-letter queues, and reconciliation support resilient operation. Security, privacy, observability, governance, and testing are integral design concerns rather than afterthoughts.

Key Terms

API: An interface through which software components communicate.

Contract: An explicit agreement describing data, operations, and behavioral guarantees.

REST: An architectural style using resources and uniform HTTP interactions.

GraphQL: A typed query language and runtime allowing clients to request response shapes.

RPC: Remote procedure call, which models a network interaction as a typed procedure invocation.

Webhook: An event-triggered HTTP callback sent to a subscriber.

Message broker: Infrastructure that routes and often stores messages between producers and consumers.

Queue: A broker structure that distributes work among competing consumers.

Stream: An ordered, durable sequence of records that can be replayed.

Idempotency: The property that repetition has the same intended effect as one execution.

Retry: A subsequent attempt after a failure or timeout.

Circuit breaker: A control that temporarily stops calls to an failing dependency.

Dead-letter queue: Storage for messages that repeatedly fail processing.

Backward compatibility: In common schema-registry usage, the ability of a new schema or reader to handle older data.

Forward compatibility: The ability of an older schema or reader to handle newer data.

Eventual consistency: A model in which replicas may temporarily differ but converge.

Saga: A sequence of local transactions coordinated with compensating actions.

Outbox pattern: A technique that records a business change and publication intent in one local transaction.

Consumer lag: The distance between the latest available stream position and a consumer’s processed position.

Contract test: A test verifying that a provider and consumer agree on an integration contract.

Review Questions

Short Answer Questions

1. Why does a network timeout not prove that a remote operation failed?
2. Distinguish a command from an event.
3. What problem does an idempotency key address?
4. Define eventual consistency.
5. Why should webhook receivers process events asynchronously after durable acceptance?

Long Answer Questions

1. Compare REST, GraphQL, and RPC in terms of coupling, performance, tooling, and appropriate use cases.
2. Explain backward and forward schema compatibility with examples of safe and unsafe changes.
3. Describe how exponential backoff, jitter, circuit breakers, and bulkheads work together.
4. Explain the outbox pattern and its relationship to at-least-once delivery.
5. Discuss security, privacy, governance, and observability requirements for a public API.

Analytical / Application Questions

1. Design an integration for a payment workflow in which the provider may process a charge but lose the response. Specify idempotency, retries, reconciliation, and user-visible states.
2. A stream consumer is two hours behind and messages contain personal data. Propose an operational and governance response, including lag diagnosis, replay controls, and retention considerations.
3. An organization wants to replace a REST endpoint with GraphQL while supporting existing mobile clients. Develop a migration and contract-testing strategy that controls query cost and manages deprecation.

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes forward compatibility in common schema-registry usage?

A. A new reader can read data written with an older schema.

B. An older reader can read data written with a newer schema.

C. A producer can ignore authentication errors.

D. A broker guarantees global message ordering.

Correct Answer: B. An older reader can read data written with a newer schema.

Question 2

Using D_n = min(C, b × 2^n + J_n), what is D_n when C = 2,000 ms, b = 1,000 ms, n = 1, and J_n = 300 ms?

A. 1,300 ms

B. 2,000 ms

C. 2,300 ms

D. 3,000 ms

Correct Answer: B. 2,000 ms

Question 3

Which mechanism most directly prevents a retried payment request from charging twice?

A. A longer HTTP timeout

B. An idempotency key stored with the result

C. A GraphQL query

D. A dead-letter queue alone

Correct Answer: B. An idempotency key stored with the result

Question 4

What is the principal purpose of a webhook?

A. To let a provider notify a subscriber through an HTTP callback

B. To replace all database transactions

C. To guarantee exactly-once processing

D. To provide global stream ordering

Correct Answer: A. To let a provider notify a subscriber through an HTTP callback

Question 5

Which integration is most suitable for distributing replayable, high-volume event records to multiple independent consumers?

A. A durable partitioned stream

B. A single synchronous GET request

C. A local function call

D. A browser cookie

Correct Answer: A. A durable partitioned stream

Question 6

What does the outbox pattern place in the same local transaction as a business change?

A. A DNS record

B. A publication-intent record

C. A client’s access token

D. A circuit-breaker timer

Correct Answer: B. A publication-intent record

Question 7

Why is jitter added to exponential backoff?

A. To increase payload size

B. To synchronize clients

C. To reduce synchronized retry storms

D. To eliminate the need for timeouts

Correct Answer: C. To reduce synchronized retry storms

Question 8

Which statement about a saga is correct?

A. It uses compensating business actions across local transactions.

B. It guarantees that all services share one database transaction.

C. It requires every operation to be a GraphQL mutation.

D. It eliminates the need for idempotent consumers.

Correct Answer: A. It uses compensating business actions across local transactions.

Question 9

Which measure is especially useful for identifying a delayed stream consumer?

A. CSS bundle size

B. Consumer lag

C. Number of API labels

D. Screen resolution

Correct Answer: B. Consumer lag

Question 10

A REST endpoint returns HTTP 200 for validation errors, authentication failures, and successful updates alike. What is the principal design problem?

A. It improves semantic clarity too much.

B. It makes automated handling and observability ambiguous.

C. It guarantees backward compatibility.

D. It enforces causal consistency.

Correct Answer: B. It makes automated handling and observability ambiguous.