3. HTTP, REST and API Engineering

Introduction and Context

Backend applications rarely operate in isolation. A web browser, mobile application, partner system, payment service, or internal dashboard must communicate with server-side capabilities through a defined interface. Application programming interfaces (APIs) provide that interface. In contemporary full-stack development, the quality of an API strongly influences system scalability, security, maintainability, developer experience, and organizational productivity.

HTTP is the foundation of most public and internal web APIs. REST provides a set of architectural constraints for modelling capabilities as resources accessed through standard HTTP semantics. API engineering extends beyond creating routes: it involves designing coherent resources, selecting appropriate methods and status codes, validating input, serializing output, handling errors consistently, supporting pagination and filtering, managing versions, documenting contracts, and operating the service reliably.

This chapter examines these concerns using examples applicable to Node.js, Django, and Spring Boot. The implementation syntax differs across frameworks, but the underlying engineering decisions are largely shared.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain HTTP requests, responses, methods, status codes, headers, caching, and content negotiation.

2. Design RESTful resources and select appropriate HTTP methods and status codes.

3. Apply validation, serialization, pagination, filtering, sorting, versioning, and idempotency.

4. Define consistent error contracts and produce OpenAPI documentation.

5. Evaluate REST, GraphQL, gRPC, and webhooks for different integration requirements.

6. Identify security, governance, operational, and ethical risks in API development.

7. Analyse an API design problem and justify an appropriate engineering solution.

HTTP as the Operational Foundation

HTTP follows a request–response model. A client sends a request containing a method, target URI, headers, and optionally a body. The server returns a response containing a status code, headers, and optionally a representation of a resource or an error.

For example:

```http

GET /api/v1/orders/781 HTTP/1.1

Host: shop.example.com

Accept: application/json

Authorization: Bearer <token>

```

A possible response is:

```http

HTTP/1.1 200 OK

Content-Type: application/json

ETag: "order-781-v4"

{"id":"781","status":"paid","total":149.50}

```

The method expresses the intended operation, while the status code communicates the result. Headers carry metadata such as authentication credentials, content type, caching instructions, correlation identifiers, and conditional-request information.

HTTP methods have different semantic properties. `GET` retrieves a representation and should be safe, meaning that it does not intentionally change server state. `POST` commonly creates a subordinate resource or requests an operation whose result is not known in advance. `PUT` replaces a resource representation and is normally idempotent: repeating the same request should produce the same intended state. `PATCH` performs a partial modification, although idempotency depends on the operation. `DELETE` removes a resource and is generally designed to be idempotent, even if the first request returns 204 and later requests return 404.

A useful distinction is between safety and idempotency. A safe method does not request a state change; an idempotent method may change state, but repeating the request has the same intended effect as performing it once. Idempotency does not guarantee identical response bodies or status codes.

Status codes should be selected deliberately. Common success codes include 200 OK for a successful retrieval or update, 201 Created when a resource is created, 202 Accepted when processing is asynchronous, and 204 No Content when the operation succeeds without a response representation. Client errors include 400 Bad Request for malformed input, 401 Unauthorized when authentication is absent or invalid, 403 Forbidden when the identity lacks permission, 404 Not Found when the resource is unavailable, 409 Conflict when the request conflicts with current state, 412 Precondition Failed for failed conditional requests, 422 Unprocessable Content for semantically invalid data, and 429 Too Many Requests for rate limiting. Server failures commonly use 500 or 503.

The distinction between 401 and 403 is operationally important. A 401 response indicates that the client must authenticate or correct its credentials. A 403 response indicates that the server understood the identity but will not permit the operation. APIs should avoid leaking whether sensitive resources exist when such disclosure creates a security risk.

From URLs to Resources: REST Design

REST, or Representational State Transfer, is an architectural style rather than a framework or protocol. Its central idea is that clients interact with identifiable resources through representations and uniform interface semantics. RESTful systems commonly exhibit client–server separation, stateless requests, cacheable responses, a uniform interface, layered architecture, and optional code-on-demand.

A resource should represent a meaningful domain concept rather than an implementation action. Prefer:

```text

GET /api/v1/customers/42/orders

```

over:

```text

GET /api/v1/getCustomerOrders?id=42

```

Collections and individual resources should be distinguishable. Typical patterns include `/products`, `/products/17`, and `/products/17/reviews`. Nouns are generally preferable to verbs in paths, while actions that do not map naturally to CRUD may be represented explicitly, such as `POST /payments/91/capture` or `POST /reports/generate`.

Resource design must reflect domain boundaries. A nested route is useful when the child has no meaningful independent identity, but excessive nesting can produce unwieldy URLs. `/customers/42/orders/781` may be appropriate for contextual access, while `/orders/781` is often better for direct retrieval. The choice should be consistent and should not imply authorization rules that the server does not actually enforce.

Representations should be stable and understandable. A resource may contain an identifier, attributes, links, metadata, and relationships. Internal database structures should not automatically become public API structures. Serialization is a deliberate translation between domain objects and external representations. It may hide sensitive fields, rename properties, flatten relationships, compute values, or select different views for list and detail endpoints.

For example, an order-list representation may return a summary, whereas an order-detail representation may include line items and payment status. This reduces payload size and prevents clients from becoming dependent on internal persistence design.

Validation, Serialization, and Error Contracts

Validation occurs at multiple layers. Transport validation checks content type, required headers, URI syntax, and whether the body is valid JSON. Structural validation checks types, required fields, lengths, formats, and allowed values. Domain validation checks rules such as whether an account is active or whether an item is available. Database constraints provide a final integrity boundary but should not be the only user-facing validation mechanism.

Validation should produce actionable errors without exposing implementation details. A robust error contract might be:

```json

{

"type": "https://api.example.com/problems/validation-error",

"title": "Request validation failed",

"status": 422,

"code": "VALIDATION_ERROR",

"detail": "One or more fields are invalid.",

"instance": "/api/v1/users",

"errors": [

{"field":"email","code":"INVALID_FORMAT","message":"Email is not valid."}

],

"traceId": "7f3a..."

}

```

The exact format may vary, but error responses should be consistent across endpoints. They should distinguish client-correctable errors from server failures, support machine-readable codes, and provide a correlation or trace identifier for support. Stack traces, SQL statements, access tokens, and personal data must not be returned to clients.

Serialization must address dates, decimal values, enumerations, nullability, and precision. Monetary values should not be represented using uncontrolled binary floating-point calculations. Dates should use an explicit standard such as ISO 8601, and timestamps should communicate timezone assumptions. Backward-compatible additions are usually safer than changing the meaning or type of an existing field.

Query Design: Pagination, Filtering, and Concurrency

Returning an unbounded collection is a common production defect. Pagination limits response size and server work. Offset pagination uses parameters such as `page` and `pageSize` or `offset` and `limit`:

```text

GET /api/v1/products?offset=40&limit=20

```

It is simple and supports direct page navigation, but records inserted or deleted during traversal can cause duplicates or omissions. Cursor pagination uses a server-generated position, for example:

```text

GET /api/v1/products?limit=20&cursor=eyJpZCI6MTIw...

```

It is generally more stable for large, frequently changing datasets, although it is less convenient for jumping to an arbitrary page. The API should define maximum page sizes, ordering rules, and the shape of pagination metadata. A cursor should be opaque; clients should not depend on its internal encoding.

Filtering and sorting should be constrained by an explicit allow-list. For example, `status=paid`, `createdAfter=2025-01-01`, and `sort=-createdAt` are safer than allowing arbitrary field expressions. The server must validate filter values and protect database performance through indexes, query limits, and cost controls.

Concurrency creates another design concern. A client may read a resource, another client may update it, and the first client may then overwrite the second change. ETags and conditional requests address this problem. A client can send `If-Match: "order-781-v4"`; the server updates only if the representation still has that version, otherwise returning 412. This is optimistic concurrency control.

Caching can reduce latency and infrastructure cost. `Cache-Control`, `ETag`, `Last-Modified`, and `Vary` influence intermediary and client behaviour. Sensitive or user-specific responses require careful cache directives. Caching must never permit one user to receive another user's representation.

Idempotency and Reliable Operations

Networks fail in ambiguous ways. A client may submit a payment request, experience a timeout, and be unable to determine whether the server processed it. Retrying a non-idempotent operation can create duplicate effects. An idempotency key allows the client to identify retries:

```http

POST /api/v1/payments

Idempotency-Key: 6d8e2c...

```

The server stores the key, relevant request fingerprint, resulting status, and response for an appropriate period. A repeated request with the same key and equivalent body returns the original result. Reuse of the key with a different body should be rejected. Idempotency storage must be designed for concurrency so that two simultaneous requests cannot both create the payment.

Idempotency is not a substitute for transactional integrity. The operation may involve database transactions, unique constraints, an outbox, or a workflow coordinator. For asynchronous operations, `202 Accepted` and a status resource such as `/jobs/991` may be more accurate than pretending the operation completed synchronously.

Versioning and Evolution

An API contract evolves as business requirements change. Versioning strategies include URI versioning (`/api/v1/orders`), media-type versioning through `Accept`, query-parameter versioning, and header-based approaches. URI versioning is visible and easy to operate; media-type versioning can keep resource identifiers stable but is less discoverable. No strategy removes the need for compatibility discipline.

Breaking changes include removing fields, changing types, altering meanings, making optional fields mandatory, and changing authorization behaviour. Adding an optional response field is often compatible, but clients that reject unknown fields can still fail. API governance should define a deprecation policy, support period, migration documentation, and telemetry to identify consumers of older versions. Versioning should not become an excuse to avoid good contract design.

OpenAPI as an Executable Contract

OpenAPI describes paths, operations, parameters, request bodies, responses, schemas, security schemes, and reusable components. It can support interactive documentation, client and server code generation, contract testing, validation, and review. A minimal example is:

```yaml

openapi: 3.0.3

info:

title: Orders API

version: 1.0.0

paths:

/orders/{id}:

get:

parameters:

• in: path

name: id

required: true

schema: {type: string}

responses:

'200':

description: Order retrieved

content:

application/json:

schema:

$ref: '#/components/schemas/Order'

'404':

description: Order not found

components:

schemas:

Order:

type: object

required: [id, status]

properties:

id: {type: string}

status: {type: string}

```

The specification should be treated as part of the product, not as documentation written after implementation. Teams may adopt design-first development, in which the contract is reviewed before coding, or code-first development, in which annotations generate the specification. Both can work if generated documentation is tested against actual behaviour. Contract tests should verify status codes, schemas, required fields, and error responses.

Node.js applications commonly use routing and validation libraries with OpenAPI middleware. Django projects may use Django REST Framework serializers, viewsets, and schema generation. Spring Boot applications often combine controller annotations, Bean Validation, and OpenAPI tooling. Framework features accelerate implementation, but they do not decide resource boundaries, compatibility policy, or business semantics.

Choosing an API Interaction Style

REST is a strong default for resource-oriented public APIs, browser-facing services, and integrations requiring broad tooling and human-readable messages. It benefits from HTTP caching, proxies, standard status codes, and independent client evolution.

GraphQL exposes a typed schema through which clients select fields and relationships. It can reduce over-fetching and under-fetching, especially for complex user interfaces, but introduces query-cost control, resolver performance, authorization complexity, caching challenges, and schema-governance requirements. GraphQL is suitable when clients have varied data needs and the organization can manage query complexity.

gRPC uses Protocol Buffers and HTTP/2-oriented communication. It provides efficient binary serialization, generated clients, streaming, and strong contracts, making it attractive for internal microservice communication and high-throughput systems. It may be less convenient for browser-native clients, casual integration partners, and environments requiring simple inspection with standard HTTP tools.

Webhooks are server-to-server callbacks triggered by events. They are useful when a client should be notified about an event without polling, such as a payment provider reporting settlement. A reliable webhook design requires signed payloads, event identifiers, retries with backoff, duplicate handling, delivery timestamps, observability, and a receiver that acknowledges quickly before processing asynchronously. Webhooks are at-least-once delivery mechanisms in many practical systems; consumers must therefore be idempotent.

A system may legitimately use more than one style: REST for its public resource API, gRPC between internal services, GraphQL for a product-facing aggregation layer, and webhooks for outbound events.

Security, Governance, and Professional Responsibility

API security begins with authentication and authorization but does not end there. Access decisions should be made at the resource and operation level. Object-level authorization failures, excessive data exposure, mass assignment, injection, weak credential handling, and unrestricted resource consumption are recurring risks. Input must be validated, output minimized, secrets protected, and administrative operations audited.

Transport encryption should be used for credentials and sensitive data. Rate limiting and quotas protect availability, while pagination and query-cost limits reduce denial-of-service risk. Logs should contain method, route, status, latency, trace identifier, and safe metadata, but should exclude passwords, tokens, and unnecessary personal information.

Governance includes ownership, schema review, lifecycle management, service-level objectives, incident procedures, and change communication. Regulatory obligations may affect retention, consent, access, correction, deletion, data residency, and auditability. Developers should avoid collecting data merely because an API could collect it. The principle of data minimization improves both privacy and operational simplicity.

Illustrative Case Study: A Reliable Order API

Background

A fictional retailer, Northstar Outfitters, operates a web store and mobile application. Its first order service exposes database-shaped endpoints, returns 200 for most outcomes, accepts unrestricted query parameters, and occasionally creates duplicate orders when mobile clients retry after network timeouts.

Situation and Problem

Customer support reports duplicate charges and inconsistent order lists. Mobile developers cannot distinguish validation failures from server failures. A partner integration has also begun depending on internal fields that the database team wants to rename.

Analysis

The service has several related weaknesses: unclear resource modelling, inadequate status semantics, no idempotency mechanism, unstable serialization, and insufficient contract governance. The problem is not simply a defective SQL query; it is a failure to define an external API contract.

Approach and Solution

The team introduces `/api/v1/orders` and `/api/v1/orders/{id}`. `POST /orders` creates an order and requires an idempotency key. Successful creation returns 201 with a `Location` header. Invalid line items return a structured 422 response; unauthenticated requests return 401; attempts to access another customer's order are handled according to the organization's information-disclosure policy.

Order representations are defined independently of database tables. List responses contain identifiers, status, totals, and timestamps, while detail responses include line items. The API uses cursor pagination with a maximum page size, an allow-list of filters, and a deterministic sort order. ETags protect updates against lost changes. The OpenAPI document specifies schemas, security requirements, examples, and error responses. Contract tests run in the deployment pipeline.

The payment workflow is made asynchronous where necessary. The order is created in a pending state, a durable event is recorded through an outbox pattern, and payment processing updates the order through a controlled workflow. A repeated idempotency key returns the original creation result rather than creating a second order.

Outcome

The revised design makes client behaviour predictable, separates public contracts from persistence structures, and makes operational failures observable. It does not eliminate all failures; instead, it ensures that retries, partial processing, and invalid requests have defined behaviour.

Learning Points

The case demonstrates that production-grade API engineering combines protocol semantics, domain modelling, persistence, reliability, documentation, and governance. Correct route naming alone would not have solved the retailer's problem.

Common Mistakes and Limitations

Frequent mistakes include returning 200 for every result, exposing stack traces, using verbs for every route, accepting arbitrary filters, returning unlimited collections, confusing authentication with authorization, and documenting only successful responses. Other defects arise when API schemas mirror database tables, dates and monetary values lack clear conventions, pagination has no stable ordering, or clients are forced to interpret human-readable error messages.

REST itself does not prescribe business workflows, authentication mechanisms, event guarantees, or database transactions. HTTP caching may be unsuitable for highly sensitive or rapidly changing data. GraphQL can create expensive queries; gRPC can increase tooling and browser complexity; webhooks require consumers to handle duplicates and delayed delivery. Engineering choices must therefore be evaluated against workload, clients, organizational capability, and operational risk.

Current Developments and Best Practices

Modern API teams increasingly combine OpenAPI with automated linting, schema-diff checks, consumer-driven contract tests, distributed tracing, and metrics-based reliability management. Zero-trust architectures encourage explicit service authentication and authorization rather than reliance on network location. Event-driven systems use webhooks, queues, and event streams alongside synchronous APIs.

Artificial intelligence can assist with schema drafting, example generation, documentation search, anomaly detection, and log analysis. However, generated contracts and code require human review because an AI system may invent fields, overlook authorization rules, or encode unsafe assumptions. Sensitive API data should not be submitted to tools without appropriate governance.

A practical best-practice checklist is to define resources around domain concepts; use HTTP semantics consistently; validate at transport and domain boundaries; publish stable schemas; constrain queries; paginate collections; implement idempotency for retry-prone commands; use conditional requests where concurrency matters; standardize errors; document with OpenAPI; test compatibility; protect sensitive data; measure latency and failures; and communicate deprecations before removal.

Chapter Summary

HTTP APIs are contracts between clients and services. Effective REST design models domain resources and applies standard methods, status codes, headers, representations, and caching semantics. Production quality requires more than routing: validation, serialization, structured errors, bounded pagination, controlled filtering, optimistic concurrency, idempotency, versioning, and OpenAPI documentation are essential. REST, GraphQL, gRPC, and webhooks solve different communication problems and may coexist in one architecture. Security, privacy, observability, governance, and compatibility determine whether an API remains dependable as its users and workload grow.

Key Terms

HTTP: A protocol for communication between clients and servers on the web.

API: A defined interface through which software components exchange capabilities or data.

REST: An architectural style based on resources, representations, and a uniform interface.

Resource: A domain entity or concept identified through a URI.

Representation: A transferable format describing a resource, such as JSON.

Idempotency: The property that repeating an operation has the same intended effect as performing it once.

Safe method: An HTTP method intended not to change server state.

Status code: A numeric response classification communicating request outcome.

Header: Metadata attached to an HTTP request or response.

Serialization: Conversion of internal objects into an external data representation.

Validation: Checking whether input is structurally and semantically acceptable.

Pagination: Dividing a collection into bounded result sets.

Cursor: An opaque position used to continue traversing a collection.

ETag: A representation identifier used for caching and conditional requests.

OpenAPI: A machine-readable specification format for HTTP API contracts.

GraphQL: A typed query language and runtime allowing clients to select response fields.

gRPC: A contract-oriented remote procedure framework commonly using Protocol Buffers.

Webhook: An HTTP callback sent in response to an event.

Content negotiation: Selection of representation formats based on client and server metadata.

Rate limiting: Restricting request frequency or resource consumption.

Review Questions

Short Answer Questions

1. Distinguish between a safe HTTP method and an idempotent HTTP method.

2. What is the difference between a 401 response and a 403 response?

3. Why should collection endpoints use pagination?

4. What purpose does an ETag serve?

5. Identify two advantages of maintaining an OpenAPI specification.

Long Answer Questions

1. Explain how REST resource modelling differs from exposing database tables and implementation actions.

2. Discuss validation, serialization, and structured error contracts as components of API quality.

3. Compare offset pagination and cursor pagination, including their operational trade-offs.

4. Explain API versioning strategies and the meaning of backward compatibility.

5. Compare REST, GraphQL, gRPC, and webhooks for public and internal integrations.

Analytical / Application Questions

1. Design an API operation for submitting a payment when clients may retry after a timeout. Specify the method, URI, headers, status codes, and idempotency behaviour.

2. A client reads an order and later overwrites another user's update. Propose an HTTP-based concurrency-control solution and explain its failure response.

3. An API returns unlimited search results and accepts arbitrary sort fields. Analyse the risks and propose a safer contract.

22. Multiple Choice Questions with Answers

Question 1

A client repeats the same payment request after a timeout. Which API mechanism most directly prevents duplicate processing?

A. An Idempotency-Key whose result is stored and reused

B. A larger JSON response

C. A 404 status for every retry

D. Removing the request body

Correct Answer: A. An Idempotency-Key whose result is stored and reused

Question 2

Which response most appropriately indicates that a new order was successfully created synchronously?

A. 200 OK

B. 201 Created

C. 202 Accepted

D. 204 No Content

Correct Answer: B. 201 Created

Question 3

What is the principal weakness of offset pagination for a frequently changing collection?

A. It cannot return JSON

B. It prevents filtering

C. Insertions or deletions can cause duplicates or omissions

D. It requires gRPC

Correct Answer: C. Insertions or deletions can cause duplicates or omissions

Question 4

Which header is most directly associated with optimistic concurrency using a previously observed representation version?

A. Accept

B. If-Match

C. Content-Length

D. User-Agent

Correct Answer: B. If-Match

Question 5

Which characteristic most strongly supports choosing gRPC for internal microservice communication?

A. Human-readable URLs are mandatory

B. Clients must select arbitrary fields at runtime

C. Efficient binary contracts and generated clients are valuable

D. The system only supports HTML forms

Correct Answer: C. Efficient binary contracts and generated clients are valuable

Question 6

A request contains syntactically valid JSON but violates a business rule such as ordering an unavailable product. Which category best describes the problem?

A. Domain validation failure

B. DNS failure

C. Content negotiation success

D. Safe-method violation

Correct Answer: A. Domain validation failure

Question 7

What is a major purpose of an OpenAPI specification?

A. To replace database transactions

B. To describe API operations and schemas for documentation and testing

C. To encrypt every response automatically

D. To guarantee zero server failures

Correct Answer: B. To describe API operations and schemas for documentation and testing

Question 8

Why should webhook consumers handle duplicate event deliveries?

A. Webhooks are normally at-most-once by definition

B. Practical delivery systems commonly retry, producing at-least-once behaviour

C. HTTP does not support event identifiers

D. Duplicate handling disables authentication

Correct Answer: B. Practical delivery systems commonly retry, producing at-least-once behaviour

Question 9

Which design best limits abuse of a collection endpoint?

A. Allow arbitrary filters and unlimited page sizes

B. Remove authentication but add more fields

C. Use bounded pagination and an allow-list of filters and sorts

D. Return database errors to the client

Correct Answer: C. Use bounded pagination and an allow-list of filters and sorts

Question 10

Which statement correctly distinguishes GraphQL from a conventional REST resource API?

A. GraphQL allows clients to select fields through a typed query schema

B. GraphQL cannot represent relationships

C. REST cannot use JSON

D. GraphQL eliminates the need for authorization

Correct Answer: A. GraphQL allows clients to select fields through a typed query schema