HTTP Protocol and Semantics

Introduction

The Hypertext Transfer Protocol (HTTP) is the foundational application-layer protocol of the Web and a central communication mechanism for RESTful APIs. It defines how clients and servers exchange messages, how resources are identified, how operations are requested, how representations are transferred, and how outcomes are communicated.

HTTP is more than a collection of verbs and status codes. Its semantics establish contractual expectations between a client and a server. For example, a client can understand that a GET request is intended to retrieve a representation, that a 404 Not Found response indicates that a resource could not be located, and that an ETag can support efficient conditional retrieval. Reliable API design depends on using these meanings consistently.

This chapter examines HTTP message structure, request-response lifecycles, methods, status codes, headers, safety, idempotence, conditional requests, caching, content negotiation, HTTPS, and the evolution from HTTP/1.1 through HTTP/2 and HTTP/3.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain the structure and lifecycle of HTTP requests and responses.
• Select HTTP methods according to their defined semantics.
• Distinguish safe, idempotent, and cacheable operations.
• Choose appropriate status codes and explain their implications.
• Use headers for metadata, validation, authentication, caching, and negotiation.
• Design conditional requests using validators such as ETag and Last-Modified.
• Explain cache-control directives and common caching risks.
• Apply content negotiation correctly, including the distinction between media types and content codings.
• Describe how HTTPS provides confidentiality, integrity, and server authentication.
• Compare HTTP/1.1, HTTP/2, and HTTP/3 and evaluate their API implications.

1. HTTP as a Semantic Contract

HTTP follows a client-server model. A user agent, mobile application, browser, or another service acts as a client. An origin server stores or generates resources. Intermediaries—including reverse proxies, gateways, load balancers, and caches—may process messages between them.

A resource is an identified object or service capability. It may be a user, an order, a document, a search result, or a computational operation. A resource is not necessarily a database row. Its current representation may be serialized as JSON, XML, HTML, an image, or another format.

A URI identifies a resource. The HTTP method indicates the intended operation, while headers provide metadata and control information. The response communicates both an outcome and, when applicable, a representation of the resource.

A useful abstraction is:

The start-line is a request line for a request or a status line for a response. Header fields are name-value metadata fields. The optional content is commonly called the message content; in modern HTTP specifications, the term “content” is preferred in many contexts over the older term “entity body.”

2. The Request-Response Lifecycle

A typical lifecycle is as follows:

1. The client resolves the server name through a naming system such as DNS.
2. The client establishes a transport connection. With HTTPS, this includes negotiation of a secure session.
3. The client sends an HTTP request.
4. An intermediary may authenticate, rate-limit, route, cache, transform, or reject the request.
5. The origin server interprets the method, target, headers, and content.
6. The origin server produces an HTTP response.
7. The response travels back through the intermediary, which may store, transform, or add metadata to it.
8. The client interprets the status code, headers, and content.

Figure: A complete HTTP request and response lifecycle through an intermediary. The response path runs from the origin server back through the intermediary to the client.

The intermediary is not always present, and it may satisfy a request from its cache without contacting the origin. Nevertheless, API developers must understand both paths because proxies and gateways can affect authentication, headers, caching, compression, routing, and observability.

Message structure

A request has the conceptual form:

METHOD /resource/path?query=value HTTP-version
Header-Name: value
Another-Header: value

optional content

For example:

GET /api/books/42 HTTP/1.1
Host: example.edu
Accept: application/json
Authorization: Bearer token

A response has the conceptual form:

HTTP-version status-code reason-phrase
Header-Name: value

optional content

For example:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=60

{"id":42,"title":"Networks"}

The reason phrase is historically present in HTTP/1.1 but is not semantically significant. Software should rely on the numeric status code.

3. HTTP Methods and Their Semantics

GET

GET requests a current representation of a target resource. It should not produce a requested state change on the server. Query parameters may affect filtering, sorting, or pagination, but they should not be used to disguise destructive actions such as /delete?id=42.

POST

POST asks the target resource to process the enclosed content according to its resource-specific semantics. It is commonly used to create subordinate resources, submit commands, or initiate processing. Repeating a POST may create multiple results unless the application adds an idempotency mechanism.

PUT

PUT requests that the state of the target resource be created or replaced with the supplied representation. A successful repeated PUT has the same intended effect as one request, assuming the representation and target remain the same.

PATCH

PATCH requests partial modification of a resource. Its idempotence depends on the patch format and operation. Setting a field to a fixed value is usually idempotent; “increment balance by 10” is not inherently idempotent when repeated.

DELETE

DELETE requests removal of the target resource’s association or representation. Repeating a deletion normally leaves the resource absent, making the method idempotent in intent, although response status codes may differ between the first and later requests.

HEAD

HEAD is equivalent to GET except that the server does not send response content. It is useful for checking metadata, existence, validators, and approximate size without transferring the representation.

OPTIONS

OPTIONS asks about communication options for a target resource or server. It is widely used in Cross-Origin Resource Sharing (CORS) preflight requests.

CONNECT and TRACE

CONNECT establishes a tunnel, commonly through a proxy. TRACE can reflect a request for diagnostic purposes and is often disabled because it may expose information or enable cross-site tracing risks.

Safety and idempotence

A safe method is defined as one whose intended semantics are read-only: the client does not request a state change. Safe does not mean risk-free. A GET may trigger logging, billing for a service, or resource-intensive computation as an incidental server effect.

An idempotent method is one where multiple identical requests have the same intended effect as one request. This does not require identical responses. For example, the first DELETE might return 204 No Content, while a later one might return 404 Not Found; the resource remains absent in both cases.

Idempotence is valuable for retries. If a network failure occurs after the server processed a PUT, repeating it is generally safe. However, idempotence does not guarantee that retries are operationally harmless: a server may perform external side effects such as sending notifications each time unless the application designs those effects carefully.

4. Status Codes and Error Semantics

HTTP status codes are grouped into five classes:

Important API statuses include:

• 200 OK: successful request with a representation or result.
• 201 Created: a resource was created; a Location header commonly identifies it.
• 202 Accepted: accepted for processing, but processing is not complete.
• 204 No Content: successful processing with no response content.
• 304 Not Modified: a conditional GET or HEAD can use its stored representation.
• 400 Bad Request: malformed or invalid request syntax or framing.
• 401 Unauthorized: authentication credentials are absent or invalid; a WWW-Authenticate challenge is generally expected.
• 403 Forbidden: the server understood the request but refuses to authorize it.
• 404 Not Found: no current representation is available for the target, or the server declines to disclose its existence.
• 405 Method Not Allowed: the method is known but not supported for the target resource. An origin server generating a 405 response must generate an Allow header listing the methods currently supported by that resource.
• 409 Conflict: the request conflicts with the current resource state.
• 412 Precondition Failed: a supplied conditional request precondition evaluated false.
• 415 Unsupported Media Type: the content format is not supported.
• 422 Unprocessable Content: the syntax is valid but the instructions or content cannot be processed, commonly for validation errors.
• 429 Too Many Requests: rate limiting applies.
• 500 Internal Server Error: unexpected server failure.
• 502 Bad Gateway: a gateway received an invalid response from an upstream server.
• 503 Service Unavailable: temporary overload or maintenance; Retry-After may help clients.
• 504 Gateway Timeout: an upstream response was not received in time.

A robust API distinguishes malformed syntax (400), unsupported representation format (415), semantically invalid content (422), authentication failure (401), and authorization failure (403). These distinctions improve client behavior and operational diagnosis.

5. Headers and Representation Metadata

Headers are extensible metadata fields. Common request headers include:

• Host: authority being addressed in HTTP/1.1; HTTP/2 and HTTP/3 use the :authority pseudo-header in the protocol mapping.
• Accept: media types acceptable in the response.
• Accept-Encoding: content codings the client can decode, such as gzip or br.
• Authorization: credentials or an authentication scheme.
• If-None-Match and If-Modified-Since: validators for conditional retrieval.
• Content-Type: media type of the request content.
• Content-Length: content length when known and applicable.

Common response headers include:

• Content-Type: media type of the response content.
• Content-Encoding: representation/content coding applied to the content, such as gzip or br. It is not a transfer-coding field.
• Content-Location: identifies a URI for the representation.
• Location: identifies a newly created resource or redirect target.
• ETag: an opaque representation validator.
• Last-Modified: a timestamp validator.
• Cache-Control: caching directives.
• Vary: request fields that influenced representation selection.
• Allow: methods supported by a resource, required on an origin-generated 405 response.

Transfer-Encoding is distinct from Content-Encoding. Transfer coding concerns how the message is framed or transferred, whereas content encoding describes an encoding applied to the representation itself. In HTTP/1.1, chunked transfer coding can delimit a message whose final size was not known in advance. HTTP/2 and HTTP/3 use their own binary framing and do not use HTTP/1.1 chunked transfer coding.

6. Conditional Requests and Validators

Conditional requests avoid unnecessary transfer and help prevent lost updates.

An entity tag, or ETag, is an opaque value associated with a selected representation. A strong validator changes whenever the representation data changes in a way relevant to byte-level equivalence. A weak validator, prefixed with W/, permits less exact equivalence and is unsuitable where byte-for-byte identity is required.

A cache revalidation sequence is:

GET /api/books/42 HTTP/1.1
If-None-Match: "v17"

If the representation remains valid, the server responds:

HTTP/1.1 304 Not Modified
ETag: "v17"

The client then uses its stored representation. If it has changed, the server returns 200 OK with the new content and a new validator.

For state-changing requests, a client can prevent overwriting a changed resource:

PUT /api/books/42 HTTP/1.1
If-Match: "v17"
Content-Type: application/json

{"title":"Updated title"}

If another client changed the resource and its tag is now "v18", the precondition fails and the server returns 412 Precondition Failed. This is optimistic concurrency control.

Last-Modified and If-Modified-Since use timestamps. They are easier to generate but can have lower precision and weaker validation properties than ETags. Clock resolution, clock synchronization, and rapid successive updates limit their reliability.

7. Caching

Caching stores a response for possible reuse. It improves latency and reduces server and network load, but only when the response is correctly cacheable and safe to reuse.

Important directives include:

• max-age=seconds: response may be considered fresh for the specified period.
• no-cache: the response may be stored but must be revalidated before reuse.
• no-store: do not store the response.
• private: intended for a private cache, such as a browser, not a shared cache.
• public: permits shared caching where other conditions allow it.
• must-revalidate: stale content must not be reused without successful validation.
• s-maxage: freshness lifetime for shared caches.

A simplified freshness calculation is:

This is not a complete cache algorithm; actual freshness can involve response dates, age calculations, heuristics, revalidation, and shared-cache rules. For a response with Cache-Control: max-age=300, a cache may treat it as fresh for 300 seconds after its applicable freshness calculation begins, subject to HTTP rules and corrections for age.

Personalized or authenticated responses require special care. A response containing one user’s private data must not be stored in a shared cache for reuse by another user. APIs should use private or no-store where appropriate and should include Vary when representation selection depends on request headers.

8. Content Negotiation

Content negotiation allows a client and server to select a suitable representation. The media type describes the format and semantics of content, for example application/json or application/problem+json.

A request may state:

Accept: application/json, application/xml;q=0.8

The quality value q expresses relative preference, not a universal probability or guarantee. The server may select JSON, XML, or return 406 Not Acceptable if no acceptable representation can be produced.

Request content is controlled by Content-Type:

Content-Type: application/json

This differs from Accept, which describes preferred response formats. A server should reject an unsupported request format with 415 Unsupported Media Type.

Compression is a separate concern. A client can send:

Accept-Encoding: br, gzip

and the server may respond:

Content-Encoding: br
Vary: Accept-Encoding

Content-Encoding says that the representation has been compressed using Brotli. Vary: Accept-Encoding informs caches that different encodings may produce different valid responses. Compression must not be confused with media type negotiation or transfer coding.

9. HTTPS and Security

HTTPS is HTTP carried over a secure transport protocol, historically TLS over TCP. It provides:

1. Confidentiality: outsiders cannot normally read protected traffic.
2. Integrity: unauthorized modifications are detectable.
3. Server authentication: certificates bind a server identity to a public key under a trust model.

TLS does not automatically provide authorization, application-level identity, or protection from insecure server code. An API still needs authentication, authorization, input validation, rate controls, and secure secret management.

Best practices include using current TLS versions, validating certificates, redirecting carefully from HTTP to HTTPS, avoiding credentials in URLs, setting secure cookie attributes, and using HSTS where operationally appropriate. Certificate validation failures should not be bypassed in production clients.

10. HTTP Evolution

HTTP/1.1

HTTP/1.1 uses textual messages and persistent TCP connections. Multiple requests can share a connection, but response ordering can create head-of-line blocking at the HTTP layer. Clients often opened several connections to improve concurrency, increasing resource use.

HTTP/2

HTTP/2 uses binary framing, multiplexed streams, header compression, and one connection for many concurrent exchanges. Multiplexing reduces application-layer head-of-line blocking, although packet loss on the underlying TCP connection can still delay all streams because TCP delivers an ordered byte stream.

HTTP/2 introduces pseudo-headers such as :method, :scheme, :authority, and :path. The underlying HTTP semantics—methods, status codes, and most headers—remain substantially familiar.

HTTP/3

HTTP/3 maps HTTP semantics onto QUIC, which uses UDP as its datagram substrate and provides encrypted, multiplexed transport streams. QUIC integrates transport establishment and TLS 1.3 handshake mechanisms rather than requiring a strictly sequential “first establish QUIC, then separately establish TLS” process. In practical terms, connection establishment combines transport and cryptographic negotiation, reducing setup delay. HTTP/3 can avoid TCP’s connection-wide head-of-line blocking because loss on one QUIC stream need not block delivery of independent streams.

HTTP/3 does not make poorly designed APIs fast automatically. Payload size, database latency, cache effectiveness, authorization checks, and server architecture remain important.

11. Worked Application: A Versioned Book API

Consider an API at https://library.example/api/books.

Creation

A client sends:

POST /api/books HTTP/1.1
Content-Type: application/json
Accept: application/json

{"title":"Distributed Systems","author":"A. Researcher"}

A suitable response is:

HTTP/1.1 201 Created
Location: /api/books/731
Content-Type: application/json
ETag: "b1"

{"id":731,"title":"Distributed Systems","author":"A. Researcher"}

201 communicates creation, while Location gives the new resource’s URI.

Retrieval and revalidation

The client later sends:

GET /api/books/731 HTTP/1.1
Accept: application/json
If-None-Match: "b1"

If unchanged, the server returns 304 Not Modified. If the title changed, it returns 200 OK with a new representation and tag, such as "b2".

Concurrent update

Two editors retrieve version "b2". Editor A successfully updates the book, producing "b3". Editor B submits a PUT with If-Match: "b2". The server returns 412 Precondition Failed rather than silently overwriting Editor A’s work.

Unsupported method

If the resource supports GET and PUT but not DELETE, a DELETE request should produce:

HTTP/1.1 405 Method Not Allowed
Allow: GET, HEAD, PUT

The Allow header is required for an origin server generating this response.

12. Illustrative Case: Protecting a University Enrollment API

Background. A university exposes an enrollment API used by a web portal and mobile application.

Problem. During registration, repeated client retries created duplicate enrollment records. Users also received stale course-capacity information from a shared cache.

Analysis. The creation operation used POST without an idempotency strategy. Course availability responses were cached without adequately distinguishing user-specific data and rapidly changing capacity.

Approach. The institution introduced an idempotency key for enrollment creation. The server records the key and resulting response for a defined period, returning the same outcome when the same client retries. Course-capacity responses use short freshness periods and validators. Responses containing personal enrollment data are marked private or no-store. Updates use If-Match to prevent lost updates, and rate limits return 429 with suitable retry guidance.

Outcome. Network retries no longer create duplicate enrollments, stale data is reduced, and conflicting edits are reported instead of overwritten.

Learning points. HTTP semantics must be combined with application-level safeguards. POST is not automatically safe to retry merely because a client did not receive a response. Caching policy must reflect privacy, volatility, and representation variance.

13. Challenges, Governance, and Best Practices

HTTP permits considerable flexibility, which creates interoperability risks. Common problems include using 200 OK for every outcome, treating GET as a command, returning HTML errors to JSON clients, ignoring Allow on 405 responses, confusing 401 and 403, and compressing responses without correct Content-Encoding and Vary metadata.

Security and governance concerns include personal-data disclosure through caches, sensitive values in query strings, inadequate audit trails, excessive error detail, weak rate limiting, and inconsistent authorization at intermediary and origin layers. API specifications should document supported methods, media types, status codes, conditional behavior, authentication schemes, and caching rules.

A professional implementation should:

• Use methods according to their standardized intent.
• Return precise status codes and machine-readable error representations.
• Include Allow on origin-generated 405 responses.
• Validate both Content-Type and request content.
• Use ETags for efficient retrieval and concurrency control.
• Define retry behavior and idempotency keys for non-idempotent workflows.
• Configure cache directives according to privacy and volatility.
• Set Vary whenever representation selection depends on request fields.
• Distinguish Content-Encoding from Transfer-Encoding.
• Use HTTPS and validate certificates.
• Test behavior through intermediaries, not only through direct connections.
• Observe latency, status distributions, cache hits, retries, and protocol errors without logging secrets.

Chapter Summary

HTTP provides both a transport-independent message model and a semantic contract for APIs. Requests contain methods, targets, headers, and optional content; responses contain status codes, headers, and optional content. Methods differ in safety and idempotence, and these properties influence retries, caching, and client design.

Status codes communicate outcomes and should be selected precisely. Headers carry representation metadata, validators, authentication information, cache directives, and negotiation preferences. ETag and conditional headers support efficient retrieval and optimistic concurrency control. Caching can improve performance but must respect privacy, freshness, and variation rules.

Content negotiation distinguishes the requested media type from the representation’s content coding. Content-Encoding describes compression of the representation, whereas Transfer-Encoding concerns message transfer. HTTPS provides confidentiality, integrity, and server authentication but does not replace application authorization. HTTP/2 improves multiplexing, while HTTP/3 uses QUIC, integrating transport and TLS establishment and reducing some forms of head-of-line blocking. Correct API design applies these semantics consistently across clients, origins, and intermediaries.

Key Terms

HTTP: Application-layer protocol for exchanging messages and representations.

Resource: An identified object, service, or capability exposed through a URI.

Representation: A transferable form of a resource’s state.

Origin server: Server with authoritative responsibility for a resource.

Intermediary: Proxy, gateway, cache, or other component between client and origin.

Media type: Label describing a representation format and semantics.

Content coding: Encoding applied to representation content, such as gzip or Brotli.

Safe method: Method whose intended semantics are read-only.

Idempotent method: Method whose repeated identical application has the same intended effect as one application.

Status code: Three-digit response code communicating processing outcome.

Validator: Metadata used to determine whether a stored representation remains current.

ETag: Opaque representation validator.

Conditional request: Request whose processing depends on a stated precondition.

Cache: System that stores responses for possible reuse.

Freshness: Period during which a cached response may be reused without revalidation.

Content negotiation: Selection of a suitable representation based on client and server capabilities.

HTTPS: HTTP protected by TLS.

QUIC: Encrypted, multiplexed transport used by HTTP/3.

CORS: Browser security mechanism governing cross-origin requests.

Optimistic concurrency control: Update strategy that detects conflicts using a version or validator.

Review Questions

Short Answer Questions

1. What is the difference between a resource and its representation?
2. Define safe and idempotent HTTP methods.
3. Why is an ETag useful in a conditional GET?
4. What information does the Allow header provide in a 405 response?
5. Distinguish Accept from Content-Type.

Long Answer Questions

1. Explain the complete request-response lifecycle involving a client, intermediary, and origin server.
2. Compare GET, POST, PUT, PATCH, and DELETE in terms of semantics, safety, and idempotence.
3. Explain how HTTP caching works and discuss the risks of caching personalized responses.
4. Describe how HTTPS protects API traffic and identify protections it does not provide.
5. Compare HTTP/1.1, HTTP/2, and HTTP/3, including their approaches to multiplexing and connection establishment.

Analytical / Application Questions

1. Design a conditional update sequence for a document edited concurrently by two clients. Specify the relevant headers and expected responses.
2. An API returns compressed JSON differently depending on Accept-Encoding. Explain which headers should be used and how a shared cache should distinguish the responses.
3. A payment client times out after submitting a request. Analyze whether it should retry using POST, and propose an API-level strategy that prevents duplicate payments.

Multiple Choice Questions with Answers

Question 1

Which method is defined as safe and idempotent?

A. GET

B. POST

C. PATCH

D. CONNECT

Correct Answer: A. GET

Question 2

An origin server returns 405 Method Not Allowed. Which header must it generate?

A. Allow

B. Retry-After

C. Content-Location

D. If-Match

Correct Answer: A. Allow

Question 3

Which header carries a client’s preferred response media types?

A. Accept

B. Content-Type

C. Content-Encoding

D. Transfer-Encoding

Correct Answer: A. Accept

Question 4

What does a 304 Not Modified response normally indicate?

A. The cached representation may be reused

B. The request body has an unsupported format

C. The server created a new resource

D. Authentication is required

Correct Answer: A. The cached representation may be reused

Question 5

Which status code commonly indicates that an If-Match precondition failed?

A. 201

B. 304

C. 412

D. 503

Correct Answer: C. 412

Question 6

What does Content-Encoding describe?

A. The media type selected through negotiation

B. An encoding applied to the representation content

C. The HTTP method used for the request

D. The transport protocol version

Correct Answer: B. An encoding applied to the representation content

Question 7

Which method is not idempotent by default and is commonly used to create subordinate resources?

A. HEAD

B. PUT

C. POST

D. DELETE

Correct Answer: C. POST

Question 8

Which HTTP version maps HTTP semantics onto QUIC?

A. HTTP/1.0

B. HTTP/1.1

C. HTTP/2

D. HTTP/3

Correct Answer: D. HTTP/3

Question 9

A response contains one user’s private enrollment data. Which directive is generally most appropriate to prevent shared-cache reuse?

A. public

B. private

C. s-maxage=3600

D. immutable

Correct Answer: B. private

Question 10

What is the principal purpose of If-Match on a state-changing request?

A. To negotiate response compression

B. To prevent updates based on a stale representation

C. To select an XML response

D. To request a proxy tunnel

Correct Answer: B. To prevent updates based on a stale representation