HTTP, REST and API Design

Introduction

Backend applications rarely operate in isolation. A web browser, mobile application, payment terminal, or another service communicates with a backend through an application programming interface (API). For web APIs, the dominant foundation is HTTP, the protocol used to exchange requests and responses across the Web.

A well-designed API is more than a collection of URLs. It expresses a coherent resource model, uses HTTP semantics consistently, communicates failures clearly, supports evolution, and remains understandable to both human developers and automated tools. In an Express application, these principles influence route design, middleware, validation, database access, error handling, security, and documentation.

This chapter presents practical and conceptual foundations for designing resource-oriented APIs with Node.js and Express. It focuses on HTTP methods, status codes, headers, content negotiation, pagination, filtering, versioning, idempotency, and OpenAPI documentation.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the structure and semantics of HTTP requests and responses;
• model application data as resources and design REST-oriented routes;
• select appropriate HTTP methods and status codes;
• use headers for metadata, caching, authentication, and content negotiation;
• design pagination, filtering, sorting, and search interfaces;
• compare API versioning strategies and choose an appropriate approach;
• distinguish safe, idempotent, and non-idempotent operations;
• document an Express API using OpenAPI;
• evaluate API designs for consistency, security, usability, and maintainability.

1. HTTP as the Contract Between Client and Server

HTTP is a request–response protocol. A client sends a request to a server, and the server returns a response. HTTP is stateless: each request should contain the information needed to process it, although servers may use cookies, tokens, databases, or caches to preserve application state outside the protocol itself.

A request generally contains:

1. a method, such as GET or POST;
2. a target URI, such as /api/v1/books/42;
3. request headers;
4. an optional message body.

A response generally contains:

1. a status code, such as 200 or 404;
2. response headers;
3. an optional representation in the response body.

For example:

GET /api/v1/books/42 HTTP/1.1
Host: library.example
Accept: application/json
Authorization: Bearer token

A possible response is:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 83

{"id":42,"title":"The Pragmatic Programmer","available":true}

The URI identifies a resource, while the representation describes its current state. A resource is an abstract concept: a book, customer, order, or collection of orders. JSON is only one possible representation of that resource.

HTTP methods and their intended semantics

Safe means that the client does not request a state-changing operation. It does not mean that the server performs no internal work: a GET may create logs or update cache statistics. Idempotent means that making the same request multiple times has the same intended effect as making it once. It does not guarantee identical response bodies or prevent logging of every request.

2. Resource-Oriented API Design

REST, or Representational State Transfer, is an architectural style rather than a library or framework. Its commonly discussed constraints include client–server separation, stateless communication, cacheability, a uniform interface, layered systems, and optional code-on-demand. Most Express APIs apply selected REST principles, especially resource-oriented URLs, HTTP semantics, and stateless requests.

Naming resources

Use nouns rather than verbs in resource paths:

GET    /books
GET    /books/42
POST   /books
PUT    /books/42
PATCH  /books/42
DELETE /books/42

Avoid designing every route as an action:

GET /getBooks
POST /createBook
POST /deleteBook/42

The method communicates the operation, while the path identifies the resource. Collection paths are commonly plural nouns. A relationship can be represented hierarchically when the relationship is meaningful:

GET /authors/7/books
GET /books/42/reviews

However, excessive nesting reduces usability. If a review has its own identity and lifecycle, /reviews/913 may be preferable to /books/42/reviews/913 for updates and deletion.

Collections and individual resources

A collection and an individual member have different meanings:

• GET /books retrieves a collection, often paginated.
• GET /books/42 retrieves one book.
• POST /books asks the server to create a book within the collection.
• PUT /books/42 replaces the identified book.
• DELETE /books/42 removes it.

Creation commonly returns 201 Created and a Location header identifying the new resource:

HTTP/1.1 201 Created
Location: /api/v1/books/43
Content-Type: application/json

Express route example

import express from 'express';

const app = express();
app.use(express.json());

app.get('/api/v1/books/:id', async (req, res, next) => {
  try {
    const book = await findBookById(req.params.id);
    if (!book) {
      return res.status(404).json({
        type: 'https://api.example/errors/not-found',
        title: 'Resource not found',
        status: 404,
        detail: 'No book exists with the supplied identifier.'
      });
    }
    res.status(200).json(book);
  } catch (error) {
    next(error);
  }
});

A professional route delegates business logic to services or controllers rather than placing database logic, validation, and response formatting in one large function.

3. Status Codes and Error Design

Status codes are part of the API contract. Clients should be able to make useful decisions without parsing human-readable messages.

The distinction between 401 and 403 is important. A 401 response indicates that valid authentication credentials have not been supplied or accepted. A 403 response indicates that the server understood the identity but refuses the operation.

Consistent error representations

An API should return errors in a predictable structure. One useful convention is the problem-details format:

{
  "type": "https://api.example/errors/validation",
  "title": "Validation failed",
  "status": 422,
  "detail": "One or more fields are invalid.",
  "instance": "/api/v1/books",
  "errors": [
    { "field": "title", "message": "Title is required" }
  ]
}

Do not expose stack traces, SQL statements, access tokens, or internal hostnames in production responses. Detailed diagnostic information belongs in protected server logs, correlated using a request or trace identifier.

4. Headers and Representations

Headers carry metadata and control processing. Common request headers include:

• Accept: representations the client can receive;
• Content-Type: media type of the request body;
• Authorization: authentication credentials;
• If-None-Match and If-Match: conditional requests;
• Idempotency-Key: client-supplied key for safely retrying certain operations;
• User-Agent: client identification.

Common response headers include:

• Content-Type: representation media type;
• Location: URI of a newly created resource or redirect target;
• ETag: representation version identifier;
• Cache-Control: caching instructions;
• Allow: methods supported by a resource;
• Retry-After: suggested delay before retrying;
• X-Request-ID or a standardized trace identifier: request correlation.

Content negotiation

Content negotiation allows the client and server to agree on a representation. A client may send:

Accept: application/json

The server should return a matching media type or 406 Not Acceptable when it cannot. The request body format is governed by Content-Type; an endpoint that expects JSON should reject unsupported formats with 415 Unsupported Media Type.

For APIs that support multiple representations, the server should include Vary: Accept so caches do not confuse responses generated for different preferences. In many JSON APIs, explicitly supporting application/json and returning 406 for unsupported formats is sufficient.

5. Validation, Filtering, Sorting and Pagination

A collection endpoint must remain usable when the dataset becomes large. Returning every record is inefficient and can create memory, bandwidth, and database problems.

Validation

Validate path parameters, query parameters, headers, and body fields at the boundary. Validation should check both type and business meaning. For example, a positive integer page size may be syntactically valid, but an excessive size should be rejected or capped.

Use allow-lists for sortable and filterable fields. Never directly concatenate an untrusted query parameter into SQL. Query builders or parameterized queries should construct database statements safely.

Filtering and sorting

A simple query interface might be:

GET /books?authorId=7&available=true&sort=-publishedAt,title

Here, -publishedAt can mean descending order and title ascending order. Document the supported fields and reject unknown fields rather than silently ignoring spelling errors.

Filtering semantics should be precise. For example, priceMin may be inclusive, while a search parameter may be case-insensitive. These decisions belong in the API documentation and tests.

Offset pagination

Offset pagination is easy to understand:

GET /books?page=3&pageSize=20

If pages are one-indexed, the offset is:

Here, page is the requested page number, pageSize is the number of records per page, and offset is the number of records skipped. This formula assumes page ≥ 1 and pageSize > 0.

A response might be:

{
  "data": [],
  "pagination": {
    "page": 3,
    "pageSize": 20,
    "totalItems": 247,
    "totalPages": 13
  }
}

If totalItems is known and pageSize > 0, the number of pages is:

Offset pagination is suitable for moderate, relatively stable datasets. Its limitations include slower database scans for large offsets and inconsistent results when records are inserted or deleted between requests.

Cursor pagination

Cursor pagination returns a token representing a position in an ordered result set:

GET /books?limit=20&after=eyJwdWJsaXNoZWRBdCI6...

The cursor should be opaque: clients should not need to interpret it. The ordering must be deterministic, commonly using a timestamp plus a unique identifier as a tie-breaker. Cursor pagination performs well for feeds and changing datasets, but it is more complex and does not naturally provide a total page count.

A robust response may include:

{
  "data": [],
  "page": {
    "limit": 20,
    "nextCursor": "opaque-token",
    "hasMore": true
  }
}

Never trust a client-provided cursor without validating its structure, expiry, and scope.

6. Idempotency, Retries and Concurrency

Networks fail in ways that create uncertainty. A client may send a request, lose the connection, and not know whether the server completed it. Retrying a non-idempotent operation can create duplicate orders or payments.

For operations such as payment creation, the client can send an idempotency key:

POST /api/v1/payments
Idempotency-Key: 6f6d8b1e-...
Content-Type: application/json

The server stores the key together with the request fingerprint and resulting response. A repeated request with the same key should return the original result rather than creating a second payment. Reuse of the same key with a different payload should produce a conflict, commonly 409.

Idempotency requires careful engineering:

• keys must be scoped to a client or account;
• records need an expiration policy;
• concurrent requests with the same key must be synchronized;
• the server must distinguish an in-progress operation from a completed one;
• sensitive response data must be protected.

Idempotency is different from authentication and does not make an operation transactional by itself.

Optimistic concurrency

Two clients may retrieve the same resource and then attempt conflicting updates. An ETag can identify the representation retrieved by a client:

GET /api/v1/books/42
ETag: "book-42-v7"

The client includes the tag when updating:

If-Match: "book-42-v7"

If the resource has changed, the server returns 412 Precondition Failed rather than silently overwriting the other update. This approach supports optimistic concurrency without locking a resource for the entire editing session.

7. API Versioning and Evolution

APIs evolve because business rules, representations, and security requirements change. A breaking change can invalidate client assumptions, so versioning needs a deliberate policy.

Common strategies include:

There is no universally correct strategy. URI versioning is often practical for diploma-level projects and public APIs because it is explicit. More important than the location of the version is disciplined compatibility management.

Usually non-breaking changes include adding optional response fields, adding new endpoints, or accepting additional optional input. Potentially breaking changes include removing fields, changing field types, changing status-code behavior, altering pagination semantics, or making an optional field mandatory.

A mature policy includes deprecation notices, migration documentation, a support period, compatibility tests, and monitoring of client usage. The Deprecation and Sunset headers may communicate retirement plans where supported by the organization’s policy.

8. Security and Governance

API design and security are inseparable. Essential controls include:

• HTTPS in production;
• strong authentication and secure token handling;
• authorization checks at the resource and operation level;
• schema validation and output encoding;
• rate limiting and abuse detection;
• bounded request sizes and timeouts;
• careful CORS configuration;
• audit logging for sensitive operations;
• protection against injection, broken object-level authorization, and mass assignment.

A particularly serious error is accepting a client-supplied userId or role and copying it directly into a database record. The server should derive ownership and privilege-sensitive values from authenticated identity and authorized business rules.

Privacy principles require collecting only necessary data, limiting retention, protecting logs, and avoiding personal information in URLs where it may be recorded by browsers, proxies, and analytics systems. API documentation should state which data is collected and how it is protected.

9. OpenAPI Documentation

OpenAPI is a machine-readable description of an HTTP API. It can support human documentation, client generation, validation, testing, and governance. Documentation must describe actual behavior; an attractive but inaccurate specification is harmful.

A minimal OpenAPI document might contain:

openapi: 3.0.3
info:
  title: Library API
  version: 1.0.0
paths:
  /api/v1/books/{id}:
    get:
      summary: Retrieve a book
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: integer
      responses:
        '200':
          description: Book found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Book'
        '404':
          description: Book not found
components:
  schemas:
    Book:
      type: object
      required: [id, title]
      properties:
        id:
          type: integer
        title:
          type: string

A useful specification documents parameters, request bodies, response schemas, authentication schemes, validation constraints, examples, error responses, and pagination behavior. Reusable components prevent inconsistent definitions. Automated linting and contract tests can identify undocumented or contradictory behavior.

Documentation should answer practical questions: Which fields are required? Which status codes can occur? Is a missing resource different from an empty collection? How are dates formatted? What are the maximum page size and rate limits? How can a client safely retry a request?

10. Applied Case: Illustrative Library Lending API

Background

A college library has an existing database of books and members. It wants a Node.js/Express API for a web application and a mobile application. The first implementation exposes database table names, returns all books at once, uses 200 for every outcome, and creates duplicate loans when mobile clients retry requests.

Problem

The API is difficult to consume and unsafe under unreliable network conditions. Clients cannot distinguish invalid credentials from missing books, large searches are slow, and simultaneous updates can overwrite one another.

Analysis

The domain contains resources such as books, members, and loans. A loan is associated with a member and a book, but it also has its own status and lifecycle. Therefore, loans should be first-class resources rather than merely an action attached to a book.

Approach and solution

The redesigned routes are:

GET    /api/v1/books
GET    /api/v1/books/{id}
POST   /api/v1/loans
GET    /api/v1/loans/{id}
PATCH  /api/v1/loans/{id}
DELETE /api/v1/loans/{id}

GET /books supports q, author, available, limit, and an opaque cursor. Unknown filter fields produce 400. The default limit is 20 and the maximum is 100. A successful loan creation returns 201 and a Location header. The request requires an Idempotency-Key, and repeated submissions return the stored result. A loan update requires If-Match; a stale tag returns 412.

Errors use a consistent problem-details structure. Authentication failures return 401, while a member who is authenticated but not allowed to borrow due to account restrictions receives 403. A book that has disappeared produces 404, and an attempt to borrow an unavailable book produces 409.

The OpenAPI document defines schemas for Book, Loan, Problem, cursor pagination, bearer authentication, and all documented responses. Contract tests verify that Express routes conform to the specification.

Outcome

The mobile application can safely retry requests, search results remain bounded, clients can distinguish error conditions, and concurrent edits do not silently overwrite one another. The API is also easier to evolve because behavior and compatibility expectations are explicit.

Learning points

The case demonstrates that API design is domain modeling. Correct route names alone are insufficient. Reliable APIs combine resource identity, HTTP semantics, validation, concurrency control, error consistency, security, and documentation.

11. Common Design Failures and Best Practices

Failure: treating every request as POST

This loses HTTP semantics and makes caching, retries, and client reasoning more difficult. Use methods according to their intended meaning.

Failure: returning 200 for errors

Clients must inspect response bodies and cannot reliably apply generic handling. Use meaningful status codes and a consistent error schema.

Failure: unbounded collections

Large responses cause latency and resource exhaustion. Set defaults and maximums, and document pagination.

Failure: undocumented behavior

A client cannot depend safely on undocumented fields, error codes, or filtering rules. Generate or maintain OpenAPI documentation as part of development.

Failure: leaking internal details

Stack traces and database messages reveal implementation information. Log details securely and return safe public errors.

Failure: changing behavior without a compatibility plan

Version breaking changes, publish migration guidance, and monitor clients before retirement.

A professional workflow usually includes schema validation, automated unit and integration tests, API contract tests, security testing, structured logs, performance monitoring, and review of OpenAPI changes during code review.

Contemporary Developments

Modern API platforms increasingly combine REST endpoints with event-driven systems, webhooks, GraphQL, or asynchronous job APIs. These approaches do not eliminate HTTP fundamentals. Webhooks still need authentication, retries, idempotency, signatures, and status handling. Long-running work may use 202 Accepted and a status resource such as /jobs/abc123.

AI-assisted tools can generate route stubs, schemas, and test cases, but generated code must be reviewed for authorization, data leakage, incorrect status codes, and insecure query construction. An AI tool cannot determine the organization’s legal obligations or domain semantics without authoritative requirements.

Chapter Summary

HTTP provides the semantics through which API clients and servers communicate. Resource-oriented design uses noun-based URIs and selects methods according to their meaning. GET retrieves, POST creates or initiates non-idempotent processing, PUT replaces, PATCH partially modifies, and DELETE removes.

Status codes communicate outcomes, while headers carry metadata for representation formats, caching, authentication, conditional requests, and retries. Content negotiation uses Accept and Content-Type. Collection endpoints should validate inputs and provide bounded filtering, sorting, and pagination. Offset pagination is simple; cursor pagination is often better for large or changing datasets.

Idempotency keys make retry-sensitive operations safer, and ETag with If-Match supports optimistic concurrency. Versioning requires compatibility discipline, deprecation planning, and migration support. OpenAPI provides a machine-readable contract that improves documentation, testing, and governance. Secure, maintainable APIs also require authorization, privacy protection, rate limiting, safe error handling, and continuous testing.

Key Terms

HTTP: Protocol for transferring messages and representations across the Web.

API: A defined interface through which software systems communicate.

Resource: An identifiable domain object or concept exposed by an API.

Representation: A transferable form of a resource, such as JSON.

REST: An architectural style based on constraints including statelessness and a uniform interface.

URI: Identifier used to locate or name a resource.

Safe method: HTTP method intended not to change resource state.

Idempotent method: Method whose repeated intended effect is equivalent to one application.

Status code: Numeric response classification describing request outcome.

Header: Metadata field carried in an HTTP request or response.

Content negotiation: Process of selecting a representation format acceptable to client and server.

Pagination: Division of a collection into manageable response segments.

Cursor: Opaque position token used for cursor-based pagination.

Idempotency key: Client value allowing the server to recognize a retried operation.

ETag: Representation identifier used in caching and conditional requests.

Optimistic concurrency: Conflict-control approach that checks whether data changed before updating.

Versioning: Strategy for evolving an API while managing compatibility.

OpenAPI: Machine-readable specification format for HTTP APIs.

Problem details: Structured representation of an API error.

Authorization: Decision about whether an authenticated identity may perform an operation.

Review Questions

Short Answer Questions

1. What is the difference between a resource and its representation?
2. Distinguish the meanings of HTTP status codes 401 and 403.
3. Why should collection endpoints implement pagination?
4. What is the purpose of an ETag and If-Match request header?
5. What information does an OpenAPI document provide?

Long Answer Questions

1. Explain how HTTP methods, resource-oriented URIs, and status codes work together in a REST-oriented API.
2. Compare offset pagination and cursor pagination, including their advantages and limitations.
3. Describe how a consistent error representation improves API usability and operations.
4. Compare URI, header, and query-parameter versioning strategies.
5. Explain the design requirements for implementing idempotency keys in an Express API.

Analytical / Application Questions

1. Design routes and status-code behavior for a course-enrolment API, including collection retrieval, enrolment creation, cancellation, and conflict handling.
2. A mobile client may retry a payment request after a timeout. Propose a safe request and server-side processing strategy using idempotency and explain how duplicate payments are prevented.
3. An API currently returns all customer records from /customers. Recommend a revised interface using filtering, sorting, pagination, validation, authorization, and OpenAPI documentation. Justify each decision.

Multiple Choice Questions with Answers

Question 1

Which HTTP method is normally used to retrieve a representation without requesting a state change?

A. GET

B. POST

C. PATCH

D. DELETE

Correct Answer: A. GET

Question 2

What status code most appropriately indicates that a new resource was successfully created?

A. 200 OK

B. 201 Created

C. 204 No Content

D. 304 Not Modified

Correct Answer: B. 201 Created

Question 3

Which header identifies the media type of a request body?

A. Accept

B. ETag

C. Content-Type

D. Location

Correct Answer: C. Content-Type

Question 4

A client is authenticated but lacks permission to delete a resource. Which response is most appropriate?

A. 400 Bad Request

B. 401 Unauthorized

C. 403 Forbidden

D. 404 Not Found

Correct Answer: C. 403 Forbidden

Question 5

For one-indexed pagination, which formula calculates the offset?

A. offset = page × pageSize

B. offset = page + pageSize

C. offset = (page − 1) × pageSize

D. offset = pageSize ÷ page

Correct Answer: C. offset = (page − 1) × pageSize

Question 6

Which technique is especially useful for preventing duplicate payment creation when a client retries a request?

A. An idempotency key

B. A larger page size

C. The User-Agent header

D. A random response status

Correct Answer: A. An idempotency key

Question 7

What does an ETag commonly support?

A. Password hashing

B. Conditional requests and cache validation

C. Database schema migration

D. Query-string encryption

Correct Answer: B. Conditional requests and cache validation

Question 8

Which pagination approach generally uses an opaque token representing a position in an ordered result set?

A. Offset pagination

B. Cursor pagination

C. Status pagination

D. Header pagination

Correct Answer: B. Cursor pagination

Question 9

What is the primary purpose of OpenAPI?

A. To replace HTTPS encryption

B. To provide a machine-readable API contract

C. To store relational database rows

D. To execute JavaScript in the browser

Correct Answer: B. To provide a machine-readable API contract

Question 10

Which response indicates that a request body uses a media type the endpoint does not support?

A. 201 Created

B. 404 Not Found

C. 415 Unsupported Media Type

D. 429 Too Many Requests

Correct Answer: C. 415 Unsupported Media Type