Web APIs and Systems Integration

Introduction and Context

Modern web applications rarely operate as isolated programs. A mobile banking application communicates with identity, payment, fraud-detection, notification, and reporting services. An AI application may send data to a model-serving endpoint, retrieve documents from a search service, and publish predictions to an enterprise workflow. These interactions are enabled by application programming interfaces (APIs): agreed interfaces through which software components exchange data and invoke operations.

Web APIs use web standards—especially HTTP, URLs, JSON, TLS, and related protocols—to make distributed systems interoperable. API design is therefore not merely a matter of exposing database operations. It involves modeling resources, selecting representations, communicating success and failure precisely, controlling change, documenting behavior, securing access, and operating the interface over its entire lifecycle.

This chapter examines RESTful and GraphQL APIs, JSON, HTTP status codes, versioning, pagination, documentation, webhooks, and real-time communication. It also considers interoperability, reliability, security, governance, and the practical integration of APIs into AI and business systems.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the role and architecture of web APIs in distributed systems.

2. Design and consume a RESTful API using HTTP methods, resources, JSON, and status codes.

3. Compare REST and GraphQL and select an appropriate approach for a given problem.

4. Apply versioning, pagination, filtering, validation, and error-handling strategies.

5. Explain API documentation, webhooks, polling, server-sent events, and WebSockets.

6. Evaluate interoperability, security, observability, and API lifecycle governance.

7. Analyze an integration scenario and justify an appropriate API architecture.

1. APIs as Contracts Between Systems

An API is a contract between a provider and one or more consumers. The contract specifies what requests are accepted, how identity is established, what responses mean, and how failures are represented. A useful API contract has at least five dimensions:

• Syntax: URL structure, HTTP methods, headers, and body format.
• Semantics: the meaning of each operation and field.
• Constraints: validation rules, authorization requirements, and rate limits.
• Failure behavior: status codes, error formats, and retry guidance.
• Evolution rules: compatibility policy and deprecation procedures.

An API differs from an implementation. A provider may replace its database or programming language without affecting consumers if the contract remains stable. This separation is central to modularity and interoperability.

Important terminology

A producer or provider exposes an API; a consumer invokes it. An endpoint is an address and operation through which a service can be reached. A resource is a conceptual entity, such as a customer or order. A representation is a transferable form of a resource, commonly JSON. A payload is the data carried in a request or response body. An API gateway is an intermediary that can route requests, authenticate clients, enforce policies, and collect metrics.

An API may be internal, partner-facing, or public. Public availability does not imply unrestricted access: authentication, authorization, quotas, and legal terms still apply.

2. HTTP and RESTful Design

REST, or Representational State Transfer, is an architectural style associated with constraints for distributed systems. A RESTful design commonly uses:

1. Client–server separation: user-interface concerns and data-service concerns are separated.

2. Stateless requests: each request contains the information needed to process it; the server does not rely on hidden conversational state between requests.

3. Cacheability: responses indicate whether they may be reused.

4. Uniform interface: resources are identified consistently, representations are manipulated through standard operations, and messages are self-descriptive.

5. Layered system: clients need not know whether they communicate directly with the origin service or through gateways and proxies.

6. Code on demand (optional): a server may provide executable code, although this is uncommon in most JSON APIs.

REST is not synonymous with “JSON over HTTP.” An API can use HTTP and JSON while violating important REST principles, for example by exposing action-heavy URLs and relying on server-side session state.

Resources and HTTP methods

A resource-oriented API uses nouns in paths:

```text

GET /v1/books # list books

POST /v1/books # create a book

GET /v1/books/42 # retrieve book 42

PATCH /v1/books/42 # partially update book 42

DELETE /v1/books/42 # delete book 42

```

`GET` retrieves a representation and should be safe: repeating it should not intentionally change server state. `PUT` replaces a resource representation and is generally intended to be idempotent, meaning that repeating the same request has the same intended effect as sending it once. `PATCH` applies a partial modification; whether a particular PATCH is idempotent depends on its definition. `DELETE` requests deletion and is normally designed to be idempotent, even if the first request returns success and later requests report that the resource is absent. `POST` commonly creates a subordinate resource or invokes an operation and is not inherently idempotent.

Idempotency matters when clients retry after a timeout. For payment or order creation, a provider can accept an Idempotency-Key request header and store the result associated with that key. A repeated request with the same key can then return the original result rather than creating a duplicate transaction.

JSON representations

JSON is a text-based data interchange format containing objects, arrays, strings, numbers, booleans, and `null`. For example:

```json

{

"id": "b-42",

"title": "Distributed Systems",

"authors": ["A. Rao", "M. Chen"],

"available": true

}

```

JSON is widely supported and readable, but it does not by itself define field meanings, units, validation rules, or version compatibility. A field called `amount` is ambiguous unless the contract states its currency and numeric representation. Dates should use a documented standard such as an ISO 8601 representation, and systems should distinguish absent fields from explicit `null` where that distinction affects meaning.

The `Content-Type` header describes the request or response body, for example `application/json`. The `Accept` header expresses formats the client can receive. Providers should validate input, reject unknown or unsafe values when appropriate, and avoid returning sensitive fields merely because they exist internally.

HTTP status codes

Status codes communicate broad outcomes:

• 200 OK: successful request with a response representation.
• 201 Created: a resource was created; the `Location` header may identify it.
• 202 Accepted: accepted for asynchronous processing, not necessarily completed.
• 204 No Content: successful operation with no response body.
• 400 Bad Request: malformed or invalid request syntax.
• 401 Unauthorized: authentication is absent or invalid. The name is historically misleading; it concerns authentication.
• 403 Forbidden: the server understood the request but refuses authorization.
• 404 Not Found: the target resource is unavailable or intentionally not disclosed.
• 409 Conflict: request conflicts with current resource state.
• 422 Unprocessable Content: syntactically valid content fails semantic validation; its use should be consistent within the API.
• 429 Too Many Requests: rate limit exceeded, often with `Retry-After`.
• 500 Internal Server Error: unexpected server-side failure.
• 502 Bad Gateway and 503 Service Unavailable: gateway or temporary service failures.

A successful HTTP status does not guarantee that a business process has completed. For instance, `202` may mean that a job was queued. Error responses should have a predictable structure, such as:

```json

{

"type": "https://api.example.org/problems/validation",

"title": "Validation failed",

"status": 422,

"detail": "One or more fields are invalid",

"errors": [{"field": "email", "message": "Invalid format"}]

}

```

A standardized problem structure helps clients handle errors without parsing human prose.

3. Designing a REST API

A disciplined design process begins with domain analysis rather than database tables. Identify resources, their relationships, lifecycle states, and permitted transitions. Then define representations and operations. For example, an order may move from `pending` to `paid`, `fulfilled`, and `cancelled`, but the API should prevent an invalid transition such as cancelling an already shipped order.

A design should specify:

• canonical URLs and identifiers;
• required and optional fields;
• validation and defaulting rules;
• authorization for each operation;
• status codes and error schema;
• concurrency behavior;
• pagination and sorting rules;
• rate limits and retry guidance.

Filtering, sorting, and pagination

Collections can become too large to return in one response. Query parameters support controlled selection:

```text

GET /v1/orders?status=paid&sort=-created_at&limit=25

```

The provider must define allowed filters and sort fields rather than translating arbitrary query text into a database query. This reduces security risk and makes performance predictable.

Offset pagination uses parameters such as `limit` and `offset`:

```text

GET /v1/orders?limit=25&offset=50

```

It is simple and useful for stable, small-to-medium datasets. However, inserts or deletions between requests can cause duplicates or omissions, and large offsets may be inefficient.

Cursor pagination returns a token representing a position in an ordered result set:

```json

{

"data": [{"id": "o91"}],

"next_cursor": "eyJjcmVhdGVkX2F0Ijoi..."

}

```

The client sends the cursor in the next request. Cursor pagination is often more stable for changing or large datasets, but cursors must be treated as opaque, may expire, and require a defined ordering. A robust ordering usually includes a unique tie-breaker, such as `(created_at, id)`, so records with equal timestamps are ordered deterministically.

Pagination should also communicate whether more results exist. Returning an unbounded collection is a common availability and denial-of-service risk.

Concurrency and caching

Two clients may retrieve the same resource, modify it, and overwrite each other. HTTP conditional requests help detect this. A server can return an `ETag`, a token representing the current representation. A client then sends `If-Match: <etag>` with an update. If the representation changed, the server can return `412 Precondition Failed` rather than silently losing an update. `If-None-Match` supports cache validation and may produce `304 Not Modified`.

Caching can reduce latency and provider load, but cache-control must reflect data sensitivity and freshness. Private or personal data should not be placed in shared caches without careful controls.

4. GraphQL

GraphQL is a query language and execution model in which a client requests a precisely shaped data graph. A GraphQL schema defines types, fields, arguments, and operations. A typical request is:

```graphql

query BookDetails($id: ID!) {

book(id: $id) {

id

title

authors { name }

}

}

```

The server returns a corresponding structure:

```json

{

"data": {

"book": {

"id": "b-42",

"title": "Distributed Systems",

"authors": [{"name": "A. Rao"}]

}

}

}

```

GraphQL commonly exposes queries for reads, mutations for state changes, and subscriptions for ongoing event streams. A response may contain both `data` and `errors`, so consumers must inspect both rather than relying only on HTTP status.

GraphQL can reduce over-fetching and under-fetching, especially for user interfaces requiring related data. It also provides a typed schema and an introspection ecosystem. However, client-selected queries create governance challenges. Deep or expensive queries can consume excessive resources; providers therefore use depth or complexity limits, persisted queries, pagination, authorization at field level, and response-size controls. GraphQL does not automatically solve caching, security, or database performance.

REST is often preferable when resources map naturally to HTTP caching, independent endpoints, simple integrations, or file transfers. GraphQL is attractive when clients have diverse data-shaping needs and a connected domain model. A hybrid architecture is reasonable: REST for commands and operational resources, GraphQL for read-oriented aggregation, or an API gateway that presents different interfaces to different consumers.

5. Versioning and API Evolution

An API evolves as requirements, security expectations, and data models change. A breaking change can cause an existing valid client to fail or change behavior. Examples include removing a field, changing a field’s type, altering authentication requirements, or changing the meaning of a status.

Common versioning approaches include:

• Path versioning: `/v1/customers` and `/v2/customers`; visible and easy to route, but can encourage long-lived parallel implementations.
• Query versioning: `/customers?version=2`; simple but less expressive and sometimes awkward for caching.
• Header or media-type versioning: clients request a representation version through headers; clean separation, but less visible during testing and troubleshooting.

There is no universally correct strategy. The important requirement is a clear compatibility policy. Providers should prefer additive changes—such as adding optional fields—when safe, tolerate unknown response fields in clients, and avoid assuming that JSON object field order has meaning.

Deprecation should be explicit. Providers should announce a sunset date, identify affected consumers, publish migration guidance, measure remaining usage, and provide a supported overlap period. Contract tests can verify that provider changes preserve agreed behavior. Semantic versioning may help communicate library changes, but HTTP API compatibility rules must still be defined independently.

6. API Documentation and Developer Experience

Documentation is part of the API product. An OpenAPI description can specify paths, parameters, request bodies, responses, authentication schemes, and schemas. From such a contract, teams can generate interactive reference pages, client SDKs, server stubs, and validation tests. Generated artifacts still require review: a technically valid schema can document an unusable design.

Effective documentation includes:

• authentication and authorization examples;
• complete request and response examples;
• error cases and retry behavior;
• pagination, filtering, and sorting rules;
• rate limits and quotas;
• idempotency and concurrency guidance;
• version and deprecation policy;
• a sandbox or test environment;
• changelog and support contacts.

A good quick-start path should take a developer from credentials to a successful request using realistic but non-sensitive data. Documentation should state whether examples are copyable, whether timestamps are UTC, and which fields are stable identifiers.

7. Webhooks and Real-Time Communication

A conventional client repeatedly asks for changes through polling. Polling is simple and reliable but can create unnecessary traffic and delay. A webhook reverses the direction: the consumer registers a callback URL, and the provider sends an event when something occurs.

A webhook delivery should include an event identifier, event type, creation time, resource reference or data, and a signature. The provider should sign a canonical representation using a secret shared with the consumer. The consumer verifies the signature before processing. It should respond quickly, enqueue the event, and process it asynchronously. Providers should retry temporary failures with bounded exponential backoff, while consumers must assume at-least-once delivery and deduplicate using the event identifier. Event ordering should not be assumed unless explicitly guaranteed.

A webhook endpoint is exposed to the public internet and therefore requires authentication, replay protection, payload-size limits, logging, and careful URL validation during subscription. Sensitive information should be minimized in event bodies.

Real-time protocols

Server-Sent Events (SSE) provide a one-way stream from server to browser over HTTP. They are suitable for notifications, progress updates, and dashboards. Browsers can automatically reconnect, and event identifiers can support resumption.

WebSockets provide a persistent, bidirectional connection. They support chat, collaborative editing, multiplayer interaction, and low-latency telemetry. They require connection management, heartbeat or ping mechanisms, authorization, backpressure, and scaling strategies such as a message broker or connection-aware routing.

Webhooks are event notifications between systems, whereas SSE and WebSockets are long-lived communication channels. The appropriate choice depends on latency, directionality, connection duration, reliability requirements, and infrastructure complexity.

8. Security, Reliability, and Observability

API security begins with transport protection. TLS prevents interception and tampering in transit, but it does not authorize a user. Authentication may use OAuth 2.0 flows, short-lived bearer tokens, mutual TLS, signed requests, or API keys. API keys are useful for identifying applications but are weak substitutes for user authorization and must be protected like credentials.

Authorization should follow least privilege. The server must enforce permissions on every request; hiding a button in a user interface is not an access control. Object-level authorization is particularly important: a user allowed to read one order must not gain access merely by changing an identifier in the URL. Sensitive fields should be filtered according to purpose.

Common risks include injection, broken object-level authorization, excessive data exposure, denial-of-service, replayed webhooks, insecure file uploads, and leaked secrets in logs. Rate limits should be designed around authenticated identities or applications and should return clear retry guidance. Input validation should constrain length, type, format, and permitted values.

Reliability requires timeouts, bounded retries, circuit breakers, bulkheads, and idempotent operations. A retry without an idempotency strategy can duplicate side effects. Clients should retry only failures likely to be temporary, such as some 429, 502, or 503 responses, and should use backoff with jitter. They should not blindly retry most validation or authorization errors.

Observability combines logs, metrics, and traces. Useful metrics include latency percentiles, error rate by endpoint, throughput, saturation, rate-limit rejections, webhook delivery success, and GraphQL query complexity. Correlation or trace identifiers allow a request to be followed across gateway, service, database, and downstream providers. Logs must avoid tokens, passwords, unnecessary personal data, and full payment details.

9. Interoperability and API Lifecycle Management

Interoperability has technical, semantic, organizational, and legal dimensions. Technical interoperability requires compatible protocols, encodings, authentication, and network access. Semantic interoperability requires agreement about concepts: whether `customer_id` identifies a person, account, or organization; whether a timestamp is creation time or update time; and what a status means. Organizational interoperability concerns ownership, support, service-level expectations, and change coordination.

An API lifecycle commonly includes:

1. Design: model resources, workflows, schemas, security, and failure behavior.

2. Review: evaluate usability, performance, privacy, and compatibility.

3. Implementation: build the service and automated contract tests.

4. Publication: provide documentation, credentials, examples, and support.

5. Operation: monitor reliability, security, usage, and consumer experience.

6. Evolution: add capabilities, deprecate safely, and migrate clients.

7. Retirement: communicate shutdown, export or migration options, and remove access securely.

An API catalog should record owners, data classification, consumers, dependencies, versions, service objectives, and retirement status. Governance should enable delivery rather than become a barrier. Lightweight design reviews and automated policy checks are often more effective than manual approval of every change.

10. Applied Case: Integrating an AI-Based Claims Service

The following is an illustrative case. A regional insurer wants a web application through which customers submit claims. An AI service classifies uploaded documents and estimates whether a claim requires manual review. The insurer must integrate identity, document storage, payment, notification, and audit systems.

Background and problem

The existing portal uses a monolithic database. It cannot reliably show processing progress, duplicate submissions occur after network timeouts, and partner repair shops need limited access to claim status. The insurer requires traceability and must protect personally identifiable information.

Analysis

The domain contains claims, documents, assessments, and payments. A claim is created before AI processing completes, so synchronous completion would make the user wait and would incorrectly represent the process. The integration must distinguish “request accepted” from “decision completed.” AI output is advisory and must not automatically deny a claim without a governed review process.

Approach and solution

The team designs a versioned REST API:

```text

POST /v1/claims

GET /v1/claims/{claim_id}

POST /v1/claims/{claim_id}/documents

GET /v1/claims/{claim_id}/events

```

`POST /v1/claims` accepts an idempotency key and returns `201 Created` when the claim record is created. Document processing returns `202 Accepted` and a processing status. The claim resource includes a state such as `submitted`, `processing`, `needs_review`, or `completed`. Cursor pagination is used for a customer’s claim history.

An internal event bus publishes `claim.document.processed` and `claim.assessment.updated`. A webhook notifies approved repair-shop partners of permitted status changes. Each event has a unique identifier, a version, and a signature. Partners acknowledge quickly and deduplicate later.

The API uses OAuth-based authorization and object-level checks. The AI service receives only the minimum document data required for classification. Predictions include model version, confidence information, timestamp, and a reason code suitable for human review. Audit records preserve who initiated an action, which model version produced an assessment, and which human approved the final outcome.

Outcome

Users receive immediate confirmation and later progress updates rather than repeated polling. Duplicate claim creation is prevented by idempotency keys. Partners receive controlled notifications without direct database access. Operations teams can trace a claim across the gateway, workflow service, AI service, and notification system.

Learning points

The case demonstrates that integration design must combine resource modeling, asynchronous workflows, security, observability, lifecycle management, and responsible AI. The API is not merely a transport layer; it is a governed boundary between organizational capabilities.

11. Best Practices and Limitations

Strong API practice includes designing from consumer use cases, using consistent naming and error formats, validating at the boundary, documenting examples, testing contracts, and treating observability as a requirement. APIs should expose stable domain concepts rather than leaking database schemas. Teams should establish timeouts, retry rules, quotas, and ownership before production release.

No API style is universally superior. REST can result in multiple round trips or awkward representations. GraphQL can create expensive queries and complicated caching. Webhooks can be difficult to secure and operate. Real-time connections consume resources and complicate deployment. JSON is flexible but can lack strong compile-time guarantees. Therefore, selection should be based on requirements, risk, consumer capability, data shape, latency, and operational maturity.

Contemporary systems increasingly use event-driven architectures, API gateways, service meshes, schema registries, and automated security testing. AI systems add concerns about prompt or data leakage, model versioning, reproducibility, fairness, and human oversight. API governance must include the data and model lifecycle when an endpoint exposes AI-generated decisions.

Chapter Summary

Web APIs provide contracts through which distributed systems exchange representations and invoke operations. RESTful APIs use HTTP resources, methods, status codes, caching, and stateless communication, while GraphQL lets clients request typed data shapes through a schema. JSON is a common representation but requires explicit semantics and validation. Effective APIs define errors, pagination, filtering, idempotency, concurrency, security, and compatibility behavior.

Versioning and deprecation protect consumers as systems evolve. OpenAPI and examples improve discoverability and enable tooling. Webhooks, SSE, and WebSockets support event-driven and real-time communication, but require attention to delivery guarantees, authentication, replay, ordering, and resource management. Interoperability involves technical and semantic agreement, while lifecycle management spans design, operation, evolution, and retirement. In AI-enabled systems, API design must also address model provenance, privacy, auditability, and human accountability.

Key Terms

API: A contract that enables software components to communicate.

Endpoint: A network-accessible API operation at a defined address.

Resource: A conceptual entity identified and manipulated through an API.

Representation: A transferable form of a resource, such as JSON.

REST: An architectural style based on constraints including statelessness and a uniform interface.

GraphQL: A typed query language and execution model for requesting data graphs.

Idempotency: The property that repeating an operation has the same intended effect as performing it once.

Pagination: Dividing a collection response into manageable portions.

Cursor: An opaque token representing a position in a paginated result set.

ETag: An identifier for a particular representation used in conditional requests.

Status code: An HTTP number communicating the broad outcome of a request.

Webhook: An event notification sent by one system to a consumer callback URL.

Server-Sent Events: A technology for one-way server-to-client streaming over HTTP.

WebSocket: A protocol supporting persistent, bidirectional communication.

OAuth 2.0: A framework for delegated authorization.

Rate limit: A restriction on request frequency or volume.

OpenAPI: A machine-readable description format for HTTP APIs.

Contract test: An automated test verifying agreed provider-consumer behavior.

API gateway: An intermediary that routes requests and applies cross-cutting policies.

API lifecycle: The stages from API design and publication through operation, evolution, and retirement.

Review Questions

Short Answer Questions

1. What is the difference between an API contract and an API implementation?

2. Explain why a `202 Accepted` response does not necessarily mean that processing has finished.

3. What is the purpose of an idempotency key?

4. Distinguish offset pagination from cursor pagination.

5. Why must a webhook consumer deduplicate events?

Long Answer Questions

1. Explain the principal REST constraints and discuss how they influence API design.

2. Compare REST and GraphQL in terms of data shaping, caching, governance, and operational risk.

3. Describe a responsible API versioning and deprecation strategy.

4. Explain how authentication, authorization, validation, rate limiting, and observability work together to secure an API.

5. Discuss the technical and semantic dimensions of interoperability in enterprise systems.

Analytical / Application Questions

1. Design endpoints, status codes, pagination, and error behavior for an API that manages university course registrations. Justify each decision.

2. A payment client times out after submitting a payment request and is unsure whether the provider processed it. Propose a safe retry and idempotency strategy.

3. An organization must deliver live warehouse alerts to a browser dashboard and status changes to external partners. Choose among polling, webhooks, SSE, and WebSockets for each use case and justify the combination.

Multiple Choice Questions with Answers

Question 1

Which HTTP method is generally used to retrieve a resource without intentionally changing server state?

A. GET

B. POST

C. PATCH

D. DELETE

Correct Answer: A. GET

Question 2

What does HTTP status 202 Accepted most appropriately indicate?

A. The request was rejected as invalid

B. The request was accepted for processing that may finish later

C. The resource was permanently deleted

D. The client is authenticated but unauthorized

Correct Answer: B. The request was accepted for processing that may finish later

Question 3

Which feature most directly prevents a retried order-creation request from creating a duplicate order?

A. A larger JSON payload

B. An idempotency key

C. A different HTTP media type

D. A browser cookie

Correct Answer: B. An idempotency key

Question 4

What is a principal advantage of cursor pagination over offset pagination for changing large datasets?

A. It eliminates the need for ordering

B. It always returns every record in one response

C. It can provide more stable traversal as records change

D. It requires no server-side implementation

Correct Answer: C. It can provide more stable traversal as records change

Question 5

Which response status commonly indicates that a client has exceeded a request rate limit?

A. 201

B. 301

C. 404

D. 429

Correct Answer: D. 429

Question 6

What is a characteristic feature of GraphQL?

A. Clients request a defined shape of fields from a typed schema

B. Every operation must use a separate URL for every field

C. It prohibits mutations of server data

D. It automatically guarantees unlimited query performance

Correct Answer: A. Clients request a defined shape of fields from a typed schema

Question 7

Why should webhook consumers process events asynchronously after quickly acknowledging receipt?

A. To avoid verifying signatures

B. To reduce delivery timeouts while handling work reliably

C. To guarantee that events arrive exactly once

D. To make HTTP status codes unnecessary

Correct Answer: B. To reduce delivery timeouts while handling work reliably

Question 8

Which HTTP mechanism helps prevent lost updates when two clients modify the same resource?

A. ETag with If-Match

B. Accept-Language

C. User-Agent only

D. A 404 response

Correct Answer: A. ETag with If-Match

Question 9

Which practice is an example of object-level authorization?

A. Checking whether a user may access the specific order identified in the request

B. Changing JSON indentation

C. Sorting response fields alphabetically

D. Compressing the response body

Correct Answer: A. Checking whether a user may access the specific order identified in the request

Question 10

What is semantic interoperability primarily concerned with?

A. Whether systems share the same physical server

B. Whether systems agree on the meaning of data and states

C. Whether all clients use the same programming language

D. Whether an API has exactly one endpoint

Correct Answer: B. Whether systems agree on the meaning of data and states