Web APIs and Systems Integration

Introduction

Modern web applications rarely operate as isolated programs. A browser-based client may obtain identity information from an authentication provider, retrieve payments from a financial service, publish events to a message broker, and notify a logistics platform through a webhook. These interactions are enabled by application programming interfaces (APIs): explicit contracts through which software systems exchange data and invoke capabilities.

This chapter focuses on designing and consuming web APIs, especially RESTful APIs, and on integrating heterogeneous systems safely and reliably. It examines resources, HTTP methods, status codes, representations, pagination, filtering, versioning, documentation, and error contracts. It also introduces GraphQL, WebSockets, webhooks, third-party services, messaging, resilience engineering, and integration testing.

An API is more than a URL that returns JSON. It is a sociotechnical contract involving semantics, security, operational behavior, compatibility, documentation, and governance. A technically valid endpoint can still be difficult to use if its naming is inconsistent, its errors are unclear, or its behavior changes without warning.

Learning Objectives

After completing this chapter, students should be able to:

• Model application data as API resources and select appropriate HTTP methods.
• Interpret HTTP status codes and design consistent error contracts.
• Design pagination, filtering, sorting, searching, and versioning strategies.
• Document and consume APIs using schemas, examples, and authentication mechanisms.
• Compare REST, GraphQL, WebSockets, webhooks, and messaging approaches.
• Apply resilience patterns to unreliable networks and third-party dependencies.
• Design integration tests that validate both behavior and contracts.
• Evaluate ethical, security, operational, and governance issues in systems integration.

1. APIs as Contracts Between Systems

An API contract specifies what a client may request, how it must request it, what response it can expect, and how failures are represented. A useful contract describes at least:

1. Operations: available actions and their HTTP methods.
2. Representations: JSON, XML, binary objects, or other data formats.
3. Validation: required fields, permitted ranges, formats, and constraints.
4. Errors: status codes, machine-readable error types, and human-readable messages.
5. Security: authentication, authorization, scopes, and transport protection.
6. Operational behavior: rate limits, timeouts, idempotency, and consistency expectations.
7. Compatibility: how changes are introduced and deprecated.

The contract is consumed by people and programs. Human-readable documentation supports learning and troubleshooting, while machine-readable descriptions support code generation, validation, and automated testing.

Important terminology

• Endpoint: A network-accessible operation, commonly identified by an HTTP method and URI.
• Resource: A conceptual object or collection managed by an API, such as a customer or order.
• Representation: A serialized view of a resource, such as a JSON document.
• Client: Software that sends requests to an API.
• Server: Software that receives requests and produces responses.
• Idempotent operation: An operation for which repeating the same request has the same intended effect as performing it once. HTTP idempotency concerns intended server state, not necessarily identical response metadata.
• Safe method: A method intended only to retrieve information, without requesting a state change.

2. RESTful API Design

Representational State Transfer (REST) is an architectural style associated with constraints such as client–server separation, stateless requests, cacheability, a uniform interface, layered systems, and resource representations. REST is not synonymous with “JSON over HTTP”; an API may use HTTP and JSON without being carefully RESTful.

2.1 Resource-oriented modeling

A resource should represent a meaningful domain concept rather than an implementation function. Good collection names are generally plural nouns:

GET    /api/v1/books
GET    /api/v1/books/42
POST   /api/v1/books
PATCH  /api/v1/books/42
DELETE /api/v1/books/42

Nested resources can express relationships when the relationship is central and the nesting remains manageable:

GET /api/v1/orders/812/items

Deeply nested paths often create coupling and authorization complexity. A separate endpoint such as /items?order_id=812 may be preferable when the child resource has an independent lifecycle.

Actions that do not map naturally to CRUD can be represented explicitly, but they should remain consistent and semantically justified. For example, POST /payments/91/refunds creates a refund operation, whereas an arbitrary verb-like path such as /refundPaymentNow is less uniform.

2.2 HTTP methods and semantics

PUT commonly means replacement: omitted mutable fields may be reset or rejected. PATCH communicates partial modification, but the server must define how missing, null, and unchanged values are interpreted. A POST request may be made idempotent by requiring an idempotency key, but the method itself does not guarantee idempotency.

2.3 Representations and content negotiation

Clients and servers negotiate representations using headers. Accept: application/json expresses a preferred response format, while Content-Type: application/json identifies the format of a request body. A server may respond with 406 Not Acceptable when it cannot produce an acceptable representation, or 415 Unsupported Media Type when it cannot process the request format.

JSON is common because it is broadly supported, but it has limitations: dates, decimal values, binary content, and object identity require explicit conventions. Financial amounts should not be represented ambiguously as binary floating-point values. An API may use integer minor units, such as cents, or a decimal string, and must document the choice and currency.

2.4 Status codes

Status codes communicate broad outcomes; the response body should provide additional detail.

A 401 response normally indicates that credentials are required or invalid; 403 indicates that authentication may have succeeded but authorization is insufficient. Implementations sometimes deliberately use 404 to avoid revealing whether a protected resource exists.

2.5 Error contracts

An error contract should be stable, safe to expose, and useful to both software and people. A problem-details style response might be:

{
  "type": "https://api.example.edu/problems/invalid-field",
  "title": "Validation failed",
  "status": 422,
  "detail": "One or more fields are invalid.",
  "instance": "/requests/7f2c",
  "errors": [
    {"field": "email", "code": "invalid_format", "message": "Enter a valid email address."}
  ]
}

The type and code values are more reliable for programs than the prose message. Error responses should not expose stack traces, SQL statements, access tokens, or personal data. A correlation or request identifier helps operators locate an event in logs without making the client responsible for internal diagnosis.

3. Querying Collections

Collection endpoints must define how clients obtain a manageable subset of data.

Pagination

Offset pagination is simple:

GET /products?limit=20&offset=40

It can become inconsistent when records are inserted or deleted between requests, and large offsets may be expensive in some databases. Cursor pagination uses an opaque cursor representing a position in an ordered result set:

GET /products?limit=20&after=eyJpZCI6MTAwfQ

The cursor should not be treated as a client-editable database key. The API should specify ordering, cursor expiry, maximum page size, and whether a next link is available. Stable ordering is essential; a common approach is ordering by a primary timestamp and a unique identifier as a tie-breaker.

A collection response may be:

{
  "data": [{"id": 101, "name": "Compiler Design"}],
  "pagination": {
    "limit": 20,
    "next_cursor": "opaque-value",
    "has_more": true
  }
}

Filtering and sorting should use documented, bounded parameters:

GET /orders?status=paid&created_after=2025-01-01&sort=-created_at

Do not turn arbitrary query strings into unrestricted database expressions. Whitelist fields and operators, validate types, impose limits, and consider the performance of each permitted filter. Search semantics, case sensitivity, time zones, and inclusive or exclusive boundaries must be documented.

Concurrency control

Two clients can overwrite each other if they read the same representation and subsequently update it. Conditional requests reduce this risk. A server can return an entity tag:

ETag: "order-812-v7"

The client then sends If-Match: "order-812-v7". If the resource changed, the server returns 412 Precondition Failed rather than silently overwriting the newer version. This is an optimistic concurrency-control strategy.

4. Versioning and Evolution

APIs evolve because requirements, security expectations, and data models change. A compatible change might add an optional response field, but removing a field, changing its type, altering error meaning, or changing authorization behavior can break clients.

Common versioning strategies include:

• Path versioning: /api/v1/customers—visible and straightforward, but duplicates paths.
• Query versioning: /customers?version=1—easy to route, but less conventional.
• Media-type versioning: a version in the Accept header—expressive, but less visible.
• Header versioning: a custom request header—flexible, but easy to overlook.

Versioning is not a substitute for compatibility discipline. Teams should publish a deprecation policy, provide migration documentation, measure usage of old versions, and communicate removal dates. Additive changes should remain genuinely optional; clients should tolerate unknown fields and servers should avoid relying on clients sending fields they do not understand.

5. Security and Governance

API security begins with transport and identity. Production APIs should use TLS, validate authentication credentials, enforce authorization at every sensitive operation, and apply least privilege. Authentication answers “who is this?”; authorization answers “what may this identity do?”

Common approaches include session cookies, OAuth 2.0 authorization flows, and signed tokens. A bearer token must be protected because possession is generally sufficient to use it. Tokens should have appropriate audience, issuer, expiry, and scope validation. Secrets should be stored in a secret-management system rather than source code or client-side bundles.

Other controls include:

• Input validation and output encoding.
• Rate limits and quotas applied according to identity and risk.
• Audit logs for sensitive actions.
• Protection against replay, injection, broken object-level authorization, and excessive data exposure.
• Data minimization, retention rules, and deletion procedures.
• Redaction of credentials and personal data in logs.

Governance also includes ownership, service-level objectives, data classification, incident response, accessibility, and legal obligations such as privacy regulation. A system should not expose more personal information merely because a database contains it.

6. Documentation and Developer Experience

An effective API document answers: what does this operation do, who may call it, what must be sent, what can be returned, and how can failures be handled? OpenAPI is widely used to describe HTTP APIs with paths, parameters, schemas, security requirements, and responses. Documentation should include realistic examples, authentication instructions, rate limits, pagination behavior, idempotency rules, and changelogs.

A useful workflow is contract-first design: define and review the interface before implementing it. This can reveal ambiguous names and incompatible assumptions early. Code-first generation can also be effective, provided generated documentation is reviewed for accuracy. Documentation testing should detect divergence between the published contract and the deployed service.

7. Alternative Integration Styles

GraphQL

GraphQL exposes a typed schema and lets clients request selected fields, often through a single endpoint. It can reduce over-fetching and under-fetching, and nested queries are convenient for user interfaces. However, flexible queries complicate authorization, caching, cost control, observability, and prevention of expensive nested operations. Servers should limit query depth or estimated cost where appropriate, use persisted queries for controlled clients, and ensure field-level authorization.

REST tends to make HTTP caching, resource monitoring, and independent endpoint ownership straightforward. GraphQL provides client-controlled selection and a strongly typed schema. Neither is universally superior; the choice depends on client diversity, domain boundaries, caching needs, and operational maturity.

WebSockets

WebSockets provide a long-lived, bidirectional connection. They are suitable for collaborative editing, multiplayer state, presence, and low-latency dashboards. They require connection authentication, heartbeat and timeout handling, reconnection logic, backpressure, message ordering decisions, and horizontal scaling strategies. A WebSocket is not automatically reliable: messages can be lost during disconnection unless the application defines acknowledgements, replay, or state resynchronization.

Webhooks

A webhook is an HTTP callback sent by one system when an event occurs. The receiver should authenticate the sender, verify a signature over the raw request body, reject stale timestamps where supported, record event identifiers, and process duplicates safely. The sender should retry transient failures with bounded exponential backoff and a dead-letter or operator-review mechanism.

A webhook handler should acknowledge quickly and place work on a durable queue rather than performing long business operations inside the incoming request. Since delivery is commonly at least once, consumers must be idempotent.

Messaging and event-driven integration

Message brokers decouple producers and consumers. Queues commonly distribute work so that one consumer processes each message, while publish–subscribe systems distribute an event to multiple subscribers. Important design decisions include delivery guarantees, ordering scope, retention, partitioning, acknowledgement, dead-letter handling, schema evolution, and observability.

“Exactly once” is difficult to guarantee across distributed boundaries. A practical design often uses at-least-once delivery plus an idempotency key or processed-message table. An outbox pattern addresses a common failure: a service updates its database but crashes before publishing an event. The service writes the business change and an outbox record in one local transaction; a publisher later reads the outbox and sends the event.

8. Resilience Engineering

Networks fail, dependencies slow down, credentials expire, and services return unexpected responses. Resilience is therefore part of API design rather than an optional optimization.

Timeouts and retries

Every outbound request should have a bounded timeout. A retry is appropriate only when the operation is safe to repeat or protected by an idempotency mechanism, and when the error is plausibly transient. Retrying validation errors or authorization failures wastes capacity. Exponential backoff with jitter reduces synchronized retry storms.

A conceptual backoff schedule can be written as:

Here, d_n is the delay before retry attempt n, d₀ is the initial delay, d_max is the maximum delay, and J is a random jitter value selected within a documented range. This is a policy model, not a universal law; the implementation must define the retry count, jitter distribution, and which failures qualify. Units for all delay variables are time, such as milliseconds. A system must also enforce an overall deadline so retries do not exceed the user’s acceptable latency.

Circuit breakers and bulkheads

A circuit breaker stops sending requests to a failing dependency after a threshold, allowing recovery and preventing resource exhaustion. It commonly has closed, open, and half-open states. A bulkhead isolates resources so failure in one integration does not consume all threads, connections, or queue capacity.

Fallbacks must preserve correctness. Returning stale data may be acceptable for a product catalogue but dangerous for account balances. If the system cannot safely determine the result of a payment, it should expose an explicit pending state rather than claiming success.

Observability

Logs, metrics, traces, and correlation identifiers help explain distributed behavior. Useful measures include request rate, error rate, latency percentiles, queue age, retry count, dependency availability, and saturation. Sensitive data must be excluded or redacted. A trace should show a request’s path through services without becoming a mechanism for collecting unnecessary personal information.

9. Integration Testing

Unit tests isolate a component; integration tests examine communication with another component or infrastructure dependency. Effective API testing includes:

• Schema tests: requests and responses conform to the published contract.
• Consumer-driven contract tests: consumer expectations are verified against provider behavior.
• Component tests: a service is tested with realistic databases, queues, or stubs.
• End-to-end tests: a complete user journey crosses deployed components.
• Failure tests: timeouts, malformed responses, duplicate events, rate limits, and authentication failures are exercised.
• Compatibility tests: old clients continue to work during supported version periods.

Test doubles must be used carefully. A mock that returns idealized behavior can conceal serialization, timeout, authorization, or status-code errors. Contract tests and ephemeral infrastructure, such as temporary databases or containers, provide stronger evidence. Tests should avoid dependence on shared mutable production-like data and should make retries and eventual consistency explicit.

A robust integration test might create an order, verify the 201 Created response and location, retrieve it, submit a duplicate creation request with the same idempotency key, and confirm that the server does not create a second order. It should then simulate a downstream timeout and verify the documented pending or failure behavior.

10. Applied Case: University Course Registration Integration

The following is an illustrative case, not a report of a particular institution.

Background

A university has a registration portal, a payment provider, a learning-management system (LMS), and an email service. Students must register for a course, pay any applicable fee, receive confirmation, and gain LMS access.

Problem

The initial implementation performs all operations synchronously in one HTTP request. When the payment provider is slow, students retry the browser request. Duplicate payment attempts occur, LMS enrollment sometimes fails after payment succeeds, and support staff cannot determine which system processed an action.

Analysis

The workflow crosses trust boundaries and contains operations with different completion times. Payment authorization is not equivalent to LMS enrollment. A single synchronous transaction cannot provide atomicity across all systems. The design requires explicit states, idempotency, durable events, and reconciliation.

Approach and solution

The registration API exposes POST /registrations with an idempotency key. It validates course capacity and student eligibility, creates a registration in pending_payment, and returns 202 Accepted with a registration identifier. The payment service receives a command containing the registration ID and idempotency key. Payment callbacks are authenticated and deduplicated by event ID.

After confirmed payment, the registration service writes an outbox event in the same local transaction that changes the registration to paid. A publisher sends RegistrationPaid to a broker. The LMS consumer enrolls the student and records the event ID before acknowledging it. Repeated delivery therefore does not create repeated enrollment. If the LMS is unavailable, the message remains available for retry and eventually enters a dead-letter queue for investigation.

The portal polls GET /registrations/{id} or subscribes to a user-facing update channel. It displays pending_payment, paid_pending_enrollment, enrolled, or attention_required rather than claiming completion prematurely. Correlation IDs connect portal requests, payment records, events, and LMS actions.

Outcome

The design prevents duplicate registration caused by browser retries, separates payment completion from enrollment completion, and provides operators with traceable states. It does not eliminate all failures: reconciliation remains necessary for cases such as a provider charging a card while its callback is lost.

Learning points

1. Distributed workflows should represent intermediate states explicitly.
2. Idempotency is essential when clients and brokers retry.
3. The outbox pattern reduces database-to-message publication gaps.
4. At-least-once delivery requires duplicate-safe consumers.
5. Operational visibility and reconciliation are part of correctness.

11. Contemporary Developments and Professional Practice

API platforms increasingly use automated schema validation, service meshes, managed queues, and observability platforms. AI-assisted tools can generate client code, test cases, and documentation drafts, but generated artifacts require human review. They may invent fields, overlook authorization rules, leak confidential schemas, or encode unsafe retry behavior. Teams should validate generated code against authoritative contracts and security policies.

The emerging practice of platform engineering treats API standards, identity, telemetry, deployment templates, and developer portals as shared products. This can improve consistency, but excessive centralization may slow domain teams. Good governance establishes minimum standards while allowing teams to select appropriate interaction styles.

Challenges and Limitations

REST conventions do not resolve domain ambiguity, distributed transaction problems, or authorization design. Pagination can produce changing views; caches can serve stale data; retries can duplicate effects; webhooks can arrive late or out of order; and GraphQL flexibility can create expensive queries. Third-party APIs may change terms, quotas, schemas, or availability without aligning with the consuming organization’s release schedule.

Consequently, integration architecture must include monitoring, version negotiation, fallback decisions, data reconciliation, provider evaluation, and exit strategies. A dependency should have an owner, documented assumptions, tested failure behavior, and a plan for credential rotation and replacement.

Best-Practice Checklist

• Model stable domain resources and use HTTP semantics consistently.
• Document request, response, error, security, and operational behavior.
• Validate and constrain all input, including filters and query complexity.
• Use appropriate status codes and stable machine-readable error codes.
• Define pagination ordering and cursor behavior.
• Protect updates with idempotency keys or conditional requests when needed.
• Make retries bounded, selective, and observable.
• Treat webhooks and messages as at-least-once unless stronger guarantees are proven.
• Use queues and outbox records for asynchronous workflows.
• Test contracts, compatibility, failures, and duplicate delivery.
• Minimize personal data and protect secrets throughout the integration.
• Publish deprecation policies and measure client usage before removal.

Chapter Summary

Web APIs are contracts that connect clients, services, data stores, and external providers. RESTful design models domain resources and uses HTTP methods, representations, status codes, caching, conditional requests, and documented query behavior. Pagination, filtering, versioning, and error contracts determine whether an API remains usable as its data and client population grow.

GraphQL offers client-selected fields through a typed schema; WebSockets support bidirectional real-time communication; webhooks notify systems of events; and message brokers support decoupled asynchronous processing. These styles have different operational costs and should be selected according to interaction requirements rather than fashion.

Reliable integration requires TLS, authentication, authorization, validation, rate control, timeouts, selective retries, idempotency, circuit breakers, observability, and reconciliation. Integration testing must verify contracts, compatibility, realistic dependencies, and failure behavior. Finally, API quality includes governance, privacy, documentation, accessibility, and responsible use of automation and AI.

Key Terms

API: A contract through which software systems communicate.

REST: An architectural style emphasizing resources, representations, and a uniform interface.

Resource: A domain entity or collection exposed through an API.

Representation: A serialized view of a resource.

Endpoint: A method and URI combination that provides an API operation.

Idempotency: The property that repeating an operation has the same intended state effect as performing it once.

Pagination: Dividing a collection into manageable response pages.

Cursor: An opaque position used to continue a paginated query.

Content negotiation: Selection of a representation format between client and server.

Error contract: A defined structure and meaning for API failures.

Optimistic concurrency: Updating only if a resource has not changed since it was read.

GraphQL: A typed query language and runtime in which clients select requested fields.

WebSocket: A persistent, bidirectional communication protocol over a connection.

Webhook: An HTTP callback triggered by an event in another system.

Message broker: Infrastructure that routes and stores messages between producers and consumers.

Outbox pattern: Writing an event to a local durable outbox with a business transaction before publishing it.

Circuit breaker: A resilience mechanism that temporarily stops calls to a failing dependency.

Bulkhead: Isolation of resources to limit failure propagation.

Contract testing: Testing that provider behavior satisfies consumer or schema expectations.

Rate limit: A restriction on the number or frequency of requests.

Review Questions

Short Answer Questions

1. What is the difference between authentication and authorization?
2. Why might cursor pagination be preferred to offset pagination for a changing collection?
3. Distinguish the usual meanings of HTTP status codes 401 and 403.
4. What problem does an idempotency key solve?
5. Why should a webhook receiver process events asynchronously when possible?

Long Answer Questions

1. Explain how to design a RESTful resource for orders, including methods, status codes, representations, and error responses.
2. Compare REST and GraphQL with respect to client flexibility, caching, authorization, and operational complexity.
3. Describe a resilient integration with a slow third-party service, including timeout, retry, circuit-breaker, and fallback decisions.
4. Explain the outbox pattern and show how it reduces inconsistency between a database transaction and event publication.
5. Develop an API versioning and deprecation policy for an organization with mobile and web clients.

Analytical / Application Questions

1. A client times out after submitting a payment request, but the provider may have processed it. Design a safe retry and reconciliation strategy.
2. An API returns 10,000 records and clients complain about slow responses. Propose pagination, filtering, sorting, and rate-limit rules, explaining their trade-offs.
3. A webhook is delivered three times and arrives out of order relative to another event. Design a consumer that preserves correctness and explain the data structures or state checks it requires.

Multiple Choice Questions with Answers

Question 1

Which HTTP method is generally intended for partial modification of an existing resource?

A. GET

B. POST

C. PATCH

D. HEAD

Correct Answer: C. PATCH

Question 2

A request is well formed but contains a semantically invalid field value. Which status code is most appropriate in the chapter's model?

A. 201

B. 422

C. 304

D. 502

Correct Answer: B. 422

Question 3

What is a principal advantage of cursor pagination over offset pagination for changing collections?

A. It requires no ordering

B. It can provide more stable continuation through results

C. It always returns every record in one response

D. It eliminates the need for authentication

Correct Answer: B. It can provide more stable continuation through results

Question 4

Which mechanism helps prevent a client from overwriting a resource changed by another client?

A. If-Match with an entity tag

B. Accept-Encoding

C. A larger page size

D. A WebSocket heartbeat

Correct Answer: A. If-Match with an entity tag

Question 5

Why should webhook consumers be designed to tolerate duplicate events?

A. Webhook delivery is commonly at least once

B. HTTP guarantees exactly three deliveries

C. Duplicates indicate successful authorization

D. JSON cannot represent unique identifiers

Correct Answer: A. Webhook delivery is commonly at least once

Question 6

What does the outbox pattern primarily address?

A. CSS rendering performance

B. The gap between committing a database change and publishing its event

C. Password hashing cost

D. GraphQL query syntax

Correct Answer: B. The gap between committing a database change and publishing its event

Question 7

Which practice is most appropriate for a retry policy?

A. Retry every error indefinitely

B. Retry only plausible transient failures with bounded backoff

C. Retry validation errors immediately without limits

D. Retry non-idempotent operations without protection

Correct Answer: B. Retry only plausible transient failures with bounded backoff

Question 8

What is a typical characteristic of GraphQL?

A. Clients select fields from a typed schema

B. It permits only fixed HTML responses

C. It requires one URI for every database table

D. It automatically guarantees inexpensive queries

Correct Answer: A. Clients select fields from a typed schema

Question 9

Which status code indicates that a request was accepted for asynchronous processing?

A. 202

B. 204

C. 401

D. 412

Correct Answer: A. 202

Question 10

What is the primary purpose of a circuit breaker?

A. To encrypt JSON fields

B. To stop or limit calls to a failing dependency temporarily

C. To replace all integration tests

D. To convert REST into GraphQL

Correct Answer: B. To stop or limit calls to a failing dependency temporarily