APIs and Distributed Web Services

Introduction and Context

Modern web applications rarely operate as isolated programs. A browser, mobile application, payment provider, identity platform, analytics system, and internal database may all exchange information through application programming interfaces (APIs). An API is a defined interface through which one software component requests data or functionality from another. Distributed web services extend this idea across networks, often allowing independently developed and deployed systems to cooperate.

The growth of cloud computing, mobile applications, Internet of Things devices, and platform ecosystems has made API design a central professional skill. A poorly designed API creates coupling, security weaknesses, inconsistent data, and difficult maintenance. A well-designed API provides a stable contract, predictable errors, appropriate performance, and clear documentation.

This chapter examines RESTful APIs, GraphQL, resource modelling, versioning, pagination, documentation, validation, error conventions, webhooks, microservices, and service-oriented architecture. It also evaluates the trade-offs involved in selecting among these approaches.

Learning Objectives

After completing this chapter, a learner should be able to:

1. Explain the role and architectural characteristics of web APIs.
2. Model domain objects as resources in a RESTful API.
3. Design HTTP operations, status codes, validation rules, pagination, and errors.
4. Compare REST and GraphQL for different application requirements.
5. Apply API versioning, documentation, authentication, and governance practices.
6. Explain webhooks and distinguish them from polling.
7. Compare microservices and service-oriented architectures.
8. Analyse the operational, security, ethical, and organisational consequences of distributed services.

1. API Foundations and Architectural Thinking

An API contract specifies what a client may request, how a request is represented, what response is returned, and how failures are communicated. The contract should be understandable independently of the implementation. For example, a client should not need to know whether data is stored in PostgreSQL, a document database, or a third-party service.

A web API commonly uses HTTP. The client sends a request consisting of a method, target URI, headers, and sometimes a body. The server returns a status code, headers, and optionally a representation such as JSON. The most common methods are:

A safe operation does not intentionally change server state. An idempotent operation has the same intended effect when repeated. Idempotence does not mean that every response is identical: a repeated DELETE may return 204 first and 404 later while still having the same intended final state.

REST, or Representational State Transfer, is an architectural style rather than a protocol. Its commonly cited constraints include client–server separation, stateless requests, cacheability, a uniform interface, layered systems, and optional code-on-demand. In practice, many “REST APIs” use HTTP and JSON but implement only some REST constraints. Precision in documentation is preferable to claiming compliance with every constraint when that is not true.

2. Resource Modelling for REST

The central REST design question is: what are the resources in the domain? Resources are identified by stable URIs and represented in formats such as JSON. In a university system, plausible resources include students, courses, enrolments, and invoices.

A collection and an individual resource should normally have distinct URIs:

GET /api/v1/students
GET /api/v1/students/42
POST /api/v1/students
PATCH /api/v1/students/42
DELETE /api/v1/students/42

Nouns are generally preferable to verbs in resource paths. /students/42 communicates the resource; /getStudent?id=42 exposes an implementation-oriented action. Actions that do not map naturally to CRUD can be represented explicitly, such as POST /payments/913/refunds, provided the semantics and consequences are documented.

Relationships can be expressed through nested paths or links. For example, GET /students/42/enrolments retrieves enrolments belonging to a student. Excessive nesting should be avoided because deep paths create coupling and make resources difficult to address independently. A practical rule is to nest only when the child has no meaningful identity outside its parent.

Representations should be consistent. A response might be:

{
  "id": "42",
  "name": "Amina Rao",
  "email": "[email protected]",
  "createdAt": "2025-02-15T10:30:00Z",
  "links": {
    "self": "/api/v1/students/42"
  }
}

Dates should use an unambiguous standard such as ISO 8601, and identifiers should have documented types and stability guarantees. Sensitive fields should not be returned merely because they exist in the database.

Resource modelling decisions

A domain concept may be represented as a resource, an embedded object, or a computed view. Embedding reduces round trips but can create stale or oversized representations. Separate resources support independent caching and permissions but may require more requests. The correct choice depends on ownership, lifecycle, access patterns, and consistency requirements.

A useful design process is:

1. Identify domain nouns and their ownership.
2. Define identifiers and lifecycle states.
3. Identify relationships and cardinality.
4. Select representations and permitted fields.
5. Map operations to HTTP methods.
6. Define authorization and validation rules.
7. Specify success, failure, and concurrency behaviour.

3. HTTP Semantics, Status Codes, and Errors

Status codes communicate broad outcome categories. Common choices include 200 OK for successful retrieval or update, 201 Created for successful creation, 202 Accepted for asynchronous processing, 204 No Content for successful operations without a response body, 400 Bad Request for malformed input, 401 Unauthorized when authentication is absent or invalid, 403 Forbidden when the caller is authenticated but not permitted, 404 Not Found when a resource is unavailable, 409 Conflict for a state conflict, 412 Precondition Failed for a failed conditional request, 422 Unprocessable Content for semantically invalid input, and 429 Too Many Requests for rate limiting. Server-side failures generally use 5xx responses.

An error response should be machine-readable and safe for users. A consistent format could be:

{
  "type": "https://api.example.edu/problems/validation-error",
  "title": "Request validation failed",
  "status": 422,
  "detail": "One or more fields are invalid.",
  "instance": "/api/v1/students",
  "errors": [
    { "field": "email", "code": "invalid_format" }
  ],
  "traceId": "7f3a9c"
}

Internal stack traces, SQL statements, access tokens, and personal data should never be exposed in production error responses. A trace identifier helps support staff correlate a client-visible failure with protected server logs.

Validation operates at several levels: syntax, type, business rule, authorization, and persistence constraints. For example, a syntactically valid date may still violate a rule that an enrolment cannot begin before a course exists. Validation should occur at the API boundary and again where necessary inside the domain or persistence layer; boundary validation is not a substitute for database constraints.

4. Pagination, Filtering, and Concurrency

Returning an entire collection is unsuitable for large or changing datasets. Pagination limits response size and improves latency. Offset pagination is easy to understand:

GET /students?limit=25&offset=50

It can become inconsistent when records are inserted or deleted between requests and may be slow for large offsets. Cursor pagination uses a position derived from a stable ordering:

GET /students?limit=25&after=eyJpZCI6NDJ9

The cursor should be opaque to clients. Results must use a deterministic ordering, commonly by creation time and a unique identifier as a tie-breaker. Cursor pagination is often better for feeds and frequently changing data, although it is less convenient for jumping directly to page 100.

A response should communicate pagination metadata, for example:

{
  "data": [
    { "id": "42", "name": "Amina Rao" }
  ],
  "pagination": {
    "limit": 25,
    "nextCursor": "eyJpZCI6NDJ9",
    "hasMore": true
  }
}

Filtering, sorting, and field selection should be constrained by documented allow-lists. Unrestricted query expressions may cause security vulnerabilities or expensive database operations. Maximum page sizes, rate limits, and query timeouts protect availability.

Concurrent updates require explicit policy. Optimistic concurrency can use an entity tag (ETag). A client retrieves a representation with an ETag and sends If-Match during an update. The server accepts the update only if the representation has not changed. This prevents a later writer from silently overwriting an earlier update.

5. Versioning and API Evolution

An API is a long-lived product. Versioning is not merely adding a number to a URL; it is managing compatibility over time. A change is breaking if existing clients may fail or change meaning, such as removing a field, changing a field type, altering authentication, or changing error semantics.

Common strategies include:

There is no universal best strategy. An organisation should select one, document it, and apply it consistently. Additive changes—such as adding optional response fields—are often compatible, but clients must be designed to ignore unknown fields. A deprecation policy should state notice periods, migration guidance, support dates, and the conditions for removal.

Contract testing checks that providers and consumers agree on request and response behaviour. Schema validation, integration tests, backward-compatibility checks, and consumer-driven contracts reduce the risk of accidental breaking changes.

6. REST and GraphQL

GraphQL is a query language and execution model in which clients request a specified selection of fields from a typed schema. A GraphQL service commonly exposes one endpoint, such as /graphql, and supports queries, mutations, and subscriptions.

A query might be:

query {
  student(id: "42") {
    id
    name
    enrolments {
      courseCode
      status
    }
  }
}

GraphQL can reduce over-fetching and under-fetching, especially for interfaces requiring data from related resources. Its schema provides strong discoverability and supports generated client types. However, query flexibility complicates caching, authorization, cost control, monitoring, and denial-of-service protection. A malicious or accidental deeply nested query can consume substantial resources.

REST commonly offers straightforward HTTP caching, resource-level authorization, mature tooling, and clear interaction with intermediaries. GraphQL can be advantageous when many clients have different data requirements or when a graph of relationships is central to the application. REST may be preferable for simple public resources, file transfer, conventional HTTP caching, or independently governed services. The decision should be based on access patterns rather than fashion.

GraphQL errors may appear alongside partial data, so clients must inspect both the data and errors fields. Field-level authorization and query complexity limits are essential. Persisted queries, depth limits, timeouts, and resolver batching can improve safety and performance.

7. Documentation, Discoverability, and Developer Experience

API documentation is part of the product. It should describe authentication, endpoints or schema, parameters, representations, status codes, errors, rate limits, examples, pagination, webhooks, and lifecycle policy. OpenAPI is widely used to describe HTTP APIs, while GraphQL schemas are commonly introspected and documented through schema tooling.

Good documentation answers practical questions: What is required? What is optional? What does an identifier mean? Can the operation be retried? How are dates and monetary values represented? What permissions are necessary? Examples should be executable or tested where possible. A documentation portal should clearly distinguish sandbox and production environments and should never publish real credentials.

8. Webhooks and Event-Driven Integration

A webhook is an HTTP callback sent by a provider to a consumer when an event occurs. Unlike polling, in which a consumer repeatedly asks whether something has changed, a webhook allows the provider to initiate notification.

A robust webhook design includes event identifiers, event types, timestamps, a versioned payload, signatures, retry policy, and delivery status. The receiver must authenticate the sender, verify the signature over the exact received bytes, reject stale or replayed messages where appropriate, and process events idempotently. Providers commonly retry after timeouts or 5xx responses, so duplicate delivery is normal rather than exceptional.

The receiver should acknowledge quickly and place work on a durable queue. It can then process the event asynchronously. Ordering is not always guaranteed; consumers should use event versions, sequence numbers, or reconciliation APIs where ordering matters. A dead-letter queue and operational dashboard help investigate permanently failing deliveries.

Polling remains useful when firewalls prevent inbound requests, when the provider cannot offer webhooks, or when periodic reconciliation is required. Many mature systems use both webhooks for low latency and polling for recovery.

9. Microservices and Service-Oriented Architecture

A service-oriented architecture (SOA) organises capabilities as services that communicate through defined contracts. Traditional SOA often emphasises enterprise integration, reusable services, governance, and sometimes an enterprise service bus. Microservices are a more specific approach in which small, autonomous teams own independently deployable services aligned with business capabilities.

Microservices can enable independent scaling, deployment, and technology choices. They also introduce distributed-systems difficulties: network failure, latency, partial availability, data consistency, observability, deployment coordination, and complex testing. Splitting a monolith into many services without clear boundaries often creates a distributed monolith—many deployments with the coupling of one large application and additional operational cost.

A service boundary should reflect cohesive business responsibility and data ownership. Each service should ideally control its data rather than allowing unrestricted shared-database access. Cross-service transactions may use a saga: a sequence of local transactions with compensating actions if a later step fails. A saga does not provide the same atomic guarantee as a single database transaction; it is appropriate only when the business process can tolerate intermediate states and compensation.

Service communication may be synchronous, such as HTTP or gRPC, or asynchronous, such as a message broker. Synchronous calls are easy to understand but create temporal coupling. Asynchronous messaging improves decoupling and resilience but requires handling duplicates, ordering, retries, and eventual consistency.

10. Security, Governance, and Ethics

API security begins with transport protection. HTTPS prevents passive interception and helps authenticate the server. Authentication establishes identity; authorization determines permitted actions. OAuth 2.0 is commonly used for delegated authorization, while OpenID Connect adds an identity layer. Tokens should have limited scope, lifetime, and audience. API keys may identify applications but should not be treated as a complete substitute for user authorization.

Important controls include object-level authorization, input validation, rate limiting, secure secret storage, audit logging, dependency updates, and protection against injection, replay, request forgery, and denial of service. Authorization must be checked for each object, not merely at the endpoint level: a user allowed to read one student record must not automatically read every student record.

Governance includes data classification, retention, deletion, incident response, change approval, and ownership. Privacy laws may require purpose limitation, minimisation, access controls, correction, and deletion. APIs should return only data necessary for the stated purpose. Logging must balance accountability against the risk of recording personal or confidential information.

Artificial intelligence services introduce additional concerns. An API calling a language model should define data-handling boundaries, prevent confidential prompts from being retained improperly, validate generated output, monitor hallucinations, and apply human review where decisions affect rights or safety. An AI-generated response is not automatically authoritative merely because it was delivered through a technically reliable API.

11. Applied Case: University Enrolment Platform

The following is an illustrative case.

Background

A university operates an enrolment platform used by a web portal, mobile application, learning-management system, and finance system. The original application exposes database tables directly through inconsistent endpoints. Mobile clients receive large responses, duplicate enrolment requests create financial records, and changes are difficult to release safely.

Problem

The university needs a stable API for students and partner systems. It must support enrolment, payment status, course capacity, and notifications while protecting personal information. The platform must also continue supporting an older mobile application during migration.

Analysis

The principal resources are students, courses, sections, enrolments, and payments. A student can have many enrolments; a section has a bounded capacity; a payment has a financial lifecycle. Enrolment creation is not a simple unrestricted database insert because it must check prerequisites, capacity, timetable conflicts, and payment rules.

Approach and Solution

The team introduces /api/v1 and publishes an OpenAPI contract. It uses GET /sections with cursor pagination and filtering by term and subject. It uses POST /enrolments for creation and requires an idempotency key so that a client retry does not create duplicate enrolments. The server returns 201 when an enrolment is created, 409 for a capacity or timetable conflict, and 422 for invalid input.

Payment processing is asynchronous. The enrolment service creates a pending payment request and returns 202 when external processing has begun. The payment provider sends a signed webhook containing an event identifier. The receiver stores processed event identifiers, verifies the signature, acknowledges quickly, and updates the payment state through a durable worker. The learning-management system consumes an event after confirmed enrolment.

An ETag protects administrative updates to course capacity. The old mobile application continues using version 1 while a new version adds optional fields and a better pagination model. Deprecation notices and migration examples are published before retirement.

Outcome

The new design reduces response sizes, prevents duplicate financial operations, and makes failures understandable to client developers. It also exposes operational costs: the university must operate queues, webhook monitoring, contract tests, access reviews, and incident procedures.

Learning Points

The case demonstrates that API quality is not determined by URI style alone. Resource modelling, business validation, idempotency, asynchronous processing, version policy, security, and observability must work together. Distributed architecture improves integration only when its additional complexity is deliberately managed.

12. Best Practices and Limitations

A professional API programme should:

• design from a documented contract and realistic consumer use cases;
• use consistent naming, representations, status codes, and error formats;
• impose limits on page size, query complexity, payload size, and request rate;
• make retries safe through idempotency keys or carefully defined idempotent operations;
• protect updates with conditional requests where lost updates are possible;
• test authentication, object authorization, validation, and failure paths;
• monitor latency, error rates, saturation, dependency failures, and webhook delivery;
• publish deprecation and support policies;
• maintain an inventory of services, owners, data classifications, and dependencies.

No architecture eliminates failure. REST does not guarantee good modelling, GraphQL does not automatically solve client performance, microservices do not automatically provide scalability, and webhooks do not guarantee exactly-once delivery. Engineering decisions must therefore be evaluated against business requirements, operational capability, regulatory obligations, and the skills of the development team.

Chapter Summary

APIs are contracts that enable software components to cooperate across process and network boundaries. RESTful design models domain concepts as resources and uses HTTP semantics, while GraphQL allows clients to select fields from a typed graph. Effective APIs require stable resource identifiers, consistent representations, validation, meaningful status codes, safe error responses, pagination, concurrency controls, documentation, and a deliberate versioning policy.

Webhooks provide event-driven notification but require signature verification, retries, idempotent processing, durable queues, and reconciliation. SOA and microservices organise capabilities into services, but distributed deployment introduces latency, partial failure, eventual consistency, and operational complexity. Security, privacy, governance, observability, and responsible data use are fundamental parts of API engineering rather than optional additions.

Key Terms

API: A defined interface through which software components communicate.

REST: An architectural style based on resources, representations, stateless interaction, and a uniform interface.

Resource: A domain concept identified by a URI and represented through an API.

Idempotent operation: An operation whose repeated application has the same intended final effect.

Safe method: An HTTP method intended not to modify server state.

Pagination: Dividing a collection into manageable result sets.

Cursor pagination: Pagination based on an opaque position in a stable ordering.

ETag: A representation identifier used for caching and conditional requests.

Validation: Checking syntax, types, business rules, authorization, and constraints.

GraphQL: A typed query language and execution model for APIs.

Webhook: An HTTP callback sent in response to an event.

Idempotency key: A client-supplied key allowing a server to recognise a retried operation.

Microservices: Independently deployable services aligned with business capabilities.

SOA: An architecture organising capabilities as contract-based services.

Saga: A sequence of local transactions coordinated with compensating actions.

Eventual consistency: A model in which replicas may temporarily differ but converge.

Rate limiting: Restricting request frequency to protect resources.

OpenAPI: A machine-readable description format for HTTP APIs.

Object-level authorization: Checking whether a caller may access a particular resource instance.

Review Questions

Short Answer Questions

1. What is the difference between authentication and authorization?
2. Why is GET generally described as safe and idempotent?
3. State two disadvantages of offset pagination.
4. What information should a useful API error response contain?
5. Why must webhook consumers handle duplicate deliveries?

Long Answer Questions

1. Explain how domain resources, relationships, HTTP methods, and representations combine in a RESTful API design.
2. Compare REST and GraphQL with respect to data fetching, caching, authorization, and operational risk.
3. Discuss the major strategies for API versioning and the meaning of backward compatibility.
4. Explain the benefits and costs of microservices compared with a modular monolith.
5. Describe a secure and reliable webhook architecture from event creation to final processing.

Analytical / Application Questions

1. Design an API for a library system containing members, books, loans, and reservations. Specify resource paths, methods, key status codes, pagination, and at least three validation rules.
2. A payment client times out after sending a request, but the server may have completed it. Analyse the failure and propose an idempotency and reconciliation strategy.
3. A GraphQL endpoint permits unlimited nested queries and is experiencing severe latency. Identify likely causes and propose controls involving query complexity, batching, caching, authorization, and monitoring.

Multiple Choice Questions with Answers

Question 1

Which HTTP method is normally used to retrieve a representation without intentionally changing server state?

A. GET

B. POST

C. PATCH

D. DELETE

Correct Answer: A. GET

Question 2

Which status code most appropriately indicates that a new resource was successfully created?

A. 200

B. 201

C. 204

D. 304

Correct Answer: B. 201

Question 3

What is a principal advantage of cursor pagination over offset pagination for frequently changing data?

A. It permits unrestricted database queries

B. It always supports random access to any page

C. It is less affected by insertions and deletions during traversal

D. It eliminates the need for ordering

Correct Answer: C. It is less affected by insertions and deletions during traversal

Question 4

What is the primary purpose of an idempotency key in a payment request?

A. To encrypt the payment amount

B. To identify the user’s password

C. To increase the page size

D. To prevent a retry from creating a duplicate operation

Correct Answer: D. To prevent a retry from creating a duplicate operation

Question 5

Which response status commonly indicates that an authenticated caller is not permitted to perform an action?

A. 401

B. 403

C. 404

D. 429

Correct Answer: B. 403

Question 6

Which characteristic is most strongly associated with GraphQL?

A. Clients request a specified selection of fields from a typed schema

B. Every resource must have a separate hostname

C. Only file uploads are supported

D. Caching is automatically simpler than HTTP caching

Correct Answer: A. Clients request a specified selection of fields from a typed schema

Question 7

Why should a webhook receiver process event identifiers idempotently?

A. Webhook providers can retry and deliver the same event more than once

B. HTTP signatures prevent all duplicate messages

C. It allows the receiver to ignore authentication

D. It guarantees events arrive in chronological order

Correct Answer: A. Webhook providers can retry and deliver the same event more than once

Question 8

Which mechanism helps prevent a lost update when two clients edit the same resource?

A. A larger JSON payload

B. An ETag with an If-Match condition

C. A public API key

D. A deeper URI hierarchy

Correct Answer: B. An ETag with an If-Match condition

Question 9

Which is a typical characteristic of microservices?

A. All services must share one database schema

B. Services are independently deployable and aligned with business capabilities

C. Network failures no longer occur

D. Every service must use the same programming language

Correct Answer: B. Services are independently deployable and aligned with business capabilities

Question 10

What is a major security risk of unrestricted GraphQL query depth?

A. Clients cannot select fields

B. The server may perform excessively expensive nested work

C. HTTP status codes become unavailable

D. TLS is automatically disabled

Correct Answer: B. The server may perform excessively expensive nested work