APIs and Systems Integration

Introduction

Modern applications rarely operate as isolated programs. A mobile application may obtain product information from a commerce server, authenticate users through an identity provider, send payments to a banking gateway, and receive delivery updates from a logistics platform. Application programming interfaces (APIs) provide the contracts that allow these independent systems to communicate.

This chapter focuses on RESTful APIs, the dominant integration style for many web applications. It explains HTTP methods and status codes, JSON representation, resource design, pagination, versioning, documentation, consumption, testing, and operational concerns. It also introduces GraphQL, webhooks, and real-time communication as alternatives or complements to REST.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain the purpose and structure of an API.
• Design resource-oriented REST endpoints using appropriate HTTP methods and status codes.
• Represent data safely and consistently using JSON.
• Implement pagination, filtering, sorting, validation, and versioning.
• Document, consume, and test APIs.
• Distinguish REST from GraphQL, webhooks, WebSockets, and server-sent events.
• Identify security, reliability, governance, and ethical issues in systems integration.

1. API Foundations

An API is a defined interface through which one software component requests capabilities or data from another. The calling component is the client; the component receiving and processing the request is the server. An API contract describes available operations, inputs, outputs, authentication requirements, errors, and behavioural expectations.

An API is not merely a URL. A useful contract includes:

1. Resources: business entities such as users, orders, or courses.
2. Representations: formats in which resources are exchanged, commonly JSON.
3. Operations: actions expressed through HTTP methods.
4. Constraints: validation, permissions, rate limits, and size limits.
5. Responses: status codes, headers, data, and error information.

HTTP is request–response based. A client sends a request containing a method, target URI, headers, and optionally a body. The server returns a response containing a status code, headers, and optionally a body.

Figure: The basic request–response relationship in an HTTP API.

REST principles

REST (Representational State Transfer) is an architectural style rather than a programming library. A RESTful design generally uses:

• Client–server separation: user-interface concerns and data concerns are independent.
• Stateless requests: each request contains the information needed to process it; the server does not depend on conversational client state stored between requests.
• Cacheability: responses indicate whether they may be cached.
• Uniform interface: resources, representations, standard methods, and predictable semantics are used consistently.
• Layered architecture: clients need not know whether they communicate directly with an application server, gateway, or cache.

A well-designed API uses nouns for resources, for example /orders, rather than action-heavy paths such as /getAllOrders. Actions are normally represented by HTTP methods. The URI identifies a resource; the method identifies the requested operation.

2. HTTP Methods, Status Codes, and JSON

HTTP methods

A safe method is intended not to change server state. Idempotent means that repeating the same request has the same intended effect as making it once. Idempotency does not mean that every response is identical: a repeated DELETE might return 204 No Content after the first request and 404 Not Found later, while the resource remains absent.

POST operations that may be retried, such as payment submission, should support an idempotency key. The server records the key and returns the original result for a repeated request, preventing accidental duplicate processing.

Status codes

Status codes communicate the broad result of a request.

Important distinctions include:

• 400 Bad Request: malformed syntax or invalid request structure.
• 401 Unauthorized: authentication is missing or invalid; despite its name, it concerns authentication.
• 403 Forbidden: identity is known, but access is not permitted.
• 404 Not Found: the resource does not exist or is intentionally hidden.
• 409 Conflict: the request conflicts with current state, such as a duplicate unique value.
• 422 Unprocessable Content: syntactically valid content fails domain validation.
• 429 Too Many Requests: a rate limit has been exceeded.
• 500 Internal Server Error: unexpected failure within the service.
• 503 Service Unavailable: temporary inability to serve, often with a Retry-After header.

A response should not use 200 OK for every outcome. Correct status codes help clients, monitoring systems, caches, and developers reason about failures.

JSON representation

JSON represents objects, arrays, strings, numbers, booleans, and null. For example:

{
  "id": "ord_1042",
  "customerId": "cus_77",
  "status": "paid",
  "total": 49.95,
  "currency": "USD",
  "items": [
    {"productId": "p_8", "quantity": 2}
  ]
}

The media type is usually application/json. Clients should send Content-Type: application/json when a body is JSON and may send Accept: application/json to state the desired response format. JSON has no native date, decimal, or binary type. APIs should therefore document date-time strings, currency precision, and file-upload mechanisms explicitly. Financial values require special care: binary floating-point values can introduce rounding errors, so servers commonly store minor units (for example, cents) as integers or use decimal arithmetic.

A consistent error format is valuable. One possible structure is:

{
  "error": {
    "code": "VALIDATION_FAILED",
    "message": "The request contains invalid fields.",
    "details": [
      {"field": "email", "issue": "Must be a valid email address."}
    ],
    "requestId": "req_91af"
  }
}

Error messages should help legitimate developers without exposing stack traces, passwords, database queries, or personal information.

3. Designing Resource-Oriented APIs

Suppose an education platform manages courses and enrolments. Reasonable endpoints include:

• GET /api/v1/courses
• GET /api/v1/courses/{courseId}
• POST /api/v1/courses
• PATCH /api/v1/courses/{courseId}
• DELETE /api/v1/courses/{courseId}
• GET /api/v1/courses/{courseId}/enrolments

A collection response might contain metadata:

{
  "data": [{"id": "c101", "title": "Web APIs"}],
  "pagination": {"nextCursor": "eyJpZCI6MTAxfQ==", "hasNext": true}
}

Design decisions should be explicit:

• Use stable identifiers and document their type.
• Use consistent pluralisation and casing.
• Avoid leaking database table names.
• Define whether deletion is physical or soft deletion.
• Specify whether missing fields mean “unchanged,” “empty,” or “null.”
• Validate input at the boundary and again where business rules require it.
• Return a Location header with 201 Created when a new resource has a discoverable URI.

Query parameters

Query parameters commonly express filtering, sorting, field selection, and pagination:

GET /courses?status=published&sort=-updatedAt&limit=20

The API should define allowed fields, default ordering, maximum page size, and behaviour for invalid parameters. Unbounded collection requests are dangerous because they can consume memory, bandwidth, and database capacity.

Pagination

Offset pagination is simple:

GET /orders?limit=20&offset=40

It is suitable for small or relatively stable datasets. Under concurrent inserts and deletions, however, records can shift between pages, causing duplicates or omissions. Cursor pagination uses a position derived from a stable ordering:

GET /orders?limit=20&after=cursorValue

It is usually more reliable for changing or large datasets. Cursors should be treated as opaque; clients should not parse them. Whichever approach is chosen, the API should document page size limits and navigation metadata.

4. Documentation, Consumption, and Testing

API documentation is part of the product. An OpenAPI description can specify paths, parameters, request bodies, responses, authentication schemes, and reusable schemas. Interactive documentation can help developers experiment, but generated examples must be reviewed for correctness and security.

A consumer should understand at least:

• Base URL and environment differences.
• Authentication and token expiry.
• Required headers and content types.
• Request and response schemas.
• Status codes and error format.
• Rate limits, pagination, and retry rules.
• Compatibility and deprecation policy.

A typical JavaScript request is:

const response = await fetch('/api/v1/courses?limit=10', {
  headers: { 'Accept': 'application/json' }
});

if (!response.ok) {
  const error = await response.json();
  throw new Error(error.error?.message ?? 'Request failed');
}

const result = await response.json();

Production consumers should also use timeouts, cancellation, bounded retries, and structured logging. Retries are appropriate mainly for transient failures such as 503 or selected network failures. They should use exponential backoff and jitter, and must not blindly retry non-idempotent operations.

Testing strategy

API testing operates at several levels:

1. Unit tests verify validation, authorization decisions, and domain functions.
2. Integration tests verify routing, database interaction, serialization, and status codes.
3. Contract tests check that provider and consumer expectations agree.
4. End-to-end tests exercise realistic workflows across services.
5. Load tests measure behaviour under expected and peak traffic.
6. Security tests examine authentication, authorization, injection, data exposure, and abuse controls.

A useful test matrix includes successful requests, missing fields, wrong types, invalid identifiers, duplicate operations, unauthorized users, empty collections, maximum page sizes, malformed JSON, upstream failures, and repeated requests. Tests should assert both status codes and important response structure; checking only that a request returned “something” is insufficient.

5. Versioning and Evolution

APIs evolve as business requirements change. A breaking change may remove a field, change its type, alter meaning, make an optional field required, or change error semantics. Additive changes can also break poorly written clients, so consumers should ignore unknown fields and providers should document compatibility expectations.

Common versioning approaches include:

• URI versioning: /api/v1/courses.
• Header or media-type versioning: clients request a version through headers.
• Query versioning: /courses?version=1.

URI versioning is easy to understand and operate; header approaches keep resource identifiers cleaner but require more tooling awareness. No method is universally superior. The important practices are a clear policy, a deprecation period, migration documentation, and monitoring of old clients. Versioning should not substitute for disciplined, backward-compatible design.

6. Integration Alternatives

GraphQL

GraphQL exposes a typed schema and generally a single endpoint. Clients specify the fields they need, which can reduce over-fetching and support complex screens. It provides strong schema introspection and can combine related data through resolvers. However, it introduces resolver complexity, authorization challenges at field level, query-cost risks, and caching patterns different from ordinary HTTP resource caching. GraphQL is often useful when clients have varied data requirements; REST may be simpler for straightforward resource operations.

Webhooks

A webhook is an event notification sent by one system to another, usually as an HTTP POST. For example, a payment provider may send payment.succeeded to a merchant endpoint. Webhook receivers must:

• Verify signatures and authenticate the sender.
• Record event identifiers for deduplication.
• Return a fast acknowledgement.
• Process work asynchronously where possible.
• Tolerate retries and out-of-order delivery.
• Provide replay or reconciliation mechanisms.

A webhook is not a replacement for querying state. Events can be delayed or lost, so robust systems periodically reconcile their local state with the provider.

Real-time communication

WebSockets provide bidirectional, long-lived communication and are suitable for chat, collaborative editing, and live games. Server-sent events (SSE) provide a persistent, server-to-client event stream over HTTP and are often simpler for notifications or dashboards. Polling is less efficient but easy to implement and sometimes adequate. The choice depends on latency, direction of communication, infrastructure, reconnection behaviour, and operational complexity.

7. Security, Governance, and Reliability

HTTPS is essential because it protects data in transit and helps prevent interception. Authentication identifies a caller; authorization determines what that caller may do. OAuth 2.0 and OpenID Connect are common standards for delegated authorization and identity, while API keys are more limited credentials and must be protected like passwords.

Core controls include:

• Least-privilege scopes and role-based or attribute-based authorization.
• Input validation and output filtering.
• Parameterized database queries.
• Rate limiting and abuse detection.
• Secret storage outside source code.
• Audit logs that exclude unnecessary sensitive data.
• CORS configuration appropriate to the application.
• Key rotation and credential revocation.
• Correlation or request identifiers for diagnosis.

Personal data should be collected only for legitimate purposes, retained no longer than necessary, and protected according to applicable privacy law and institutional policy. APIs can unintentionally expose sensitive relationships—for example, allowing a user to change an object identifier and view another person’s record. Object-level authorization must therefore be checked for every protected resource, not only at login.

Reliability requires timeouts, health checks, circuit breakers where appropriate, graceful degradation, observability, and documented service-level expectations. Distributed systems can fail partially: a catalogue service may respond while a payment service is unavailable. Integration design should define what happens in such cases and how incomplete workflows are recovered.

Applied Case: Illustrative Course Enrolment Integration

Background

A college’s student portal displays courses from an academic system and sends confirmed enrolments to a learning-management system (LMS). The two systems are owned by different teams and have different release schedules.

Problem

The original integration used a nightly file export. Students waited up to 24 hours for access, duplicate records appeared after file retransmission, and failures were difficult to trace.

Analysis

The teams identified three requirements: near-real-time enrolment updates, safe retries, and clear ownership of student data. The academic system remained the source of truth for enrolment status; the LMS owned learning content and access provisioning.

Approach and solution

The academic system exposed GET /api/v1/courses and GET /api/v1/enrolments/{id} for reconciliation. When an enrolment was confirmed, it published a signed webhook containing an event ID, enrolment ID, student ID, course ID, and event type. The LMS acknowledged the webhook quickly, stored the event ID, and processed it through a queue. Repeated event IDs were ignored after the first successful processing. A nightly reconciliation job compared active enrolments and repaired missed events.

The API used 201 Created for new enrolments, 409 Conflict for an already-existing enrolment, and 422 Unprocessable Content for a valid request that violated academic rules. The contract was documented with OpenAPI, and contract tests ran in both teams’ deployment pipelines.

Outcome

Access was normally provisioned within minutes rather than a day. Duplicate delivery no longer created duplicate access, and reconciliation provided recovery from outages. Operational dashboards reported webhook latency, failure rate, retry count, and reconciliation differences.

Learning points

The case illustrates that integration is more than endpoint construction. Ownership, event identity, idempotency, asynchronous processing, reconciliation, observability, and contract governance are equally important.

Challenges and Best Practices

Common integration failures include inconsistent naming, undocumented assumptions, excessive payloads, weak authorization, unlimited queries, retry storms, and treating external systems as perfectly reliable. Effective practice includes:

1. Design the contract before implementation.
2. Use standard HTTP semantics consistently.
3. Validate at boundaries and return actionable errors.
4. Make retry behaviour and idempotency explicit.
5. Paginate every potentially large collection.
6. Keep schemas backward compatible where possible.
7. Publish examples and executable contract tests.
8. Monitor latency, error classes, saturation, and dependency health.
9. Protect personal data and minimize exposure.
10. Provide reconciliation for important asynchronous workflows.

AI-assisted coding can generate client libraries, test cases, and documentation drafts, but generated code must be reviewed. Developers must not paste production secrets or personal data into external tools, and they must verify authentication, error handling, dependency licences, and data flows.

Chapter Summary

APIs define contracts through which software systems exchange capabilities and data. RESTful APIs model resources, use HTTP methods with established semantics, return meaningful status codes, and commonly exchange JSON representations. Good designs address validation, errors, pagination, filtering, authentication, authorization, documentation, testing, versioning, and operational reliability. Consumers require timeouts, safe retries, and clear handling of failures. GraphQL offers client-selected data, webhooks provide asynchronous event notifications, and WebSockets or SSE support real-time communication. Secure and reliable integration depends on least privilege, idempotency, observability, reconciliation, privacy protection, and disciplined evolution of contracts.

Key Terms

API: A defined interface through which software components communicate.

REST: An architectural style based on resources, representations, and a uniform interface.

Resource: A conceptual entity exposed by an API.

Representation: A transferable format of a resource, such as JSON.

Idempotent: Having the same intended effect when repeated.

Safe method: An HTTP method intended not to change server state.

Status code: A three-digit HTTP result indicator.

JSON: A lightweight text format for structured data exchange.

Pagination: Dividing a large collection into manageable responses.

Cursor: An opaque position used to navigate a collection.

OpenAPI: A machine-readable specification format for HTTP APIs.

Contract test: A test verifying agreement between API provider and consumer.

Webhook: An event notification delivered to an HTTP endpoint.

GraphQL: A typed query language and runtime for client-selected data.

WebSocket: A persistent, bidirectional communication protocol.

Authentication: Verification of a caller’s identity.

Authorization: Determination of permitted actions or resources.

Rate limiting: Restricting request volume over a period.

Reconciliation: Comparing systems to detect and repair state differences.

Versioning: Managing changes to an API contract over time.

Review Questions

Short Answer Questions

1. What is the difference between authentication and authorization?
2. Why is 201 Created appropriate for some POST requests?
3. Define idempotency and name two idempotent HTTP methods.
4. What problem does pagination solve?
5. State one important difference between a webhook and polling.

Long Answer Questions

1. Explain REST’s client–server, statelessness, cacheability, uniform-interface, and layered-system principles.
2. Compare offset pagination and cursor pagination, including a situation suited to each.
3. Describe a practical API documentation and testing strategy for a new service.
4. Explain how an API should evolve without unnecessarily breaking existing consumers.
5. Compare REST, GraphQL, and WebSockets in terms of data access, communication pattern, and operational concerns.

Analytical / Application Questions

1. Design endpoints and status-code behaviour for creating, listing, updating, and cancelling appointments. Justify your method choices.
2. A payment client times out after submitting a POST and cannot tell whether the payment succeeded. Propose a safe integration design using idempotency, querying, and retry rules.
3. A webhook receiver processes duplicate and out-of-order events. Design a processing and reconciliation approach that preserves correct final state.

Multiple Choice Questions with Answers

Question 1

Which HTTP method is normally used to retrieve a resource without changing server state?

A. GET

B. POST

C. PATCH

D. DELETE

Correct Answer: A. GET

Question 2

What does HTTP status code 401 most directly indicate?

A. The resource was deleted

B. Authentication is missing or invalid

C. The server completed the request successfully

D. The request exceeded its page size

Correct Answer: B. Authentication is missing or invalid

Question 3

Which design most directly prevents a retried payment request from creating a duplicate payment?

A. Using a larger JSON response

B. Using an idempotency key

C. Returning status 200 for every response

D. Disabling authentication

Correct Answer: B. Using an idempotency key

Question 4

What is a principal advantage of cursor pagination over offset pagination for a changing, large dataset?

A. It removes the need for authorization

B. It guarantees every request returns all records

C. It is less affected by records shifting between pages

D. It requires no ordering

Correct Answer: C. It is less affected by records shifting between pages

Question 5

Which status code best represents a syntactically valid request that violates a domain validation rule?

A. 201

B. 301

C. 422

D. 503

Correct Answer: C. 422

Question 6

What is the primary role of an OpenAPI document?

A. To replace HTTPS encryption

B. To describe API paths, inputs, outputs, and security requirements

C. To store production passwords

D. To provide a database engine

Correct Answer: B. To describe API paths, inputs, outputs, and security requirements

Question 7

Which statement correctly describes a webhook?

A. A client repeatedly asks for updates at fixed intervals

B. A server sends an event notification to a receiving endpoint

C. A database table used for caching

D. A method that always replaces a resource

Correct Answer: B. A server sends an event notification to a receiving endpoint

Question 8

Which technology is designed for persistent bidirectional communication?

A. WebSocket

B. HEAD

C. CSV

D. DNS

Correct Answer: A. WebSocket

Question 9

What is the main security distinction between authentication and authorization?

A. Authentication identifies a caller; authorization determines permitted actions

B. Authentication encrypts data; authorization compresses it

C. Authentication paginates data; authorization sorts it

D. They are exactly the same operation

Correct Answer: A. Authentication identifies a caller; authorization determines permitted actions

Question 10

Why should a webhook receiver record event identifiers?

A. To make JSON invalid

B. To deduplicate repeated event deliveries

C. To eliminate the need for HTTPS

D. To increase page size

Correct Answer: B. To deduplicate repeated event deliveries