Web Protocols and HTTP

Introduction

The web is a distributed information system in which clients request resources from servers using agreed communication protocols. A browser, mobile application, or backend service may act as a client; a web server, application server, or API gateway may act as a server. The protocols that connect these components determine how resources are located, requested, transferred, secured, and interpreted.

For backend developers, understanding web protocols is more important than memorising isolated commands. A reliable web service depends on the interaction of URLs, DNS, HTTP, TLS, browsers, operating systems, and application code. When a request fails, the cause may be an incorrect domain name, a refused TCP connection, an invalid HTTP method, an authentication failure, a CORS policy, or an application error. Effective troubleshooting requires a layered understanding of this process.

This chapter focuses on HTTP-based web communication and the practical foundations required to build and diagnose web services.

Learning Objectives

By the end of this chapter, you should be able to:

• interpret URLs and explain how DNS resolves domain names;
• describe the request–response model of HTTP;
• select appropriate HTTP methods and status codes;
• use headers, cookies, and sessions appropriately;
• explain content negotiation and media types;
• distinguish HTTP from HTTPS and describe the role of TLS;
• explain CORS and common cross-origin problems;
• design services that respect HTTP statelessness;
• use command-line tools and browser tools to troubleshoot web communication.

1. Locating Web Resources: URLs and DNS

A Uniform Resource Locator (URL) identifies the location and access method of a resource. A typical URL is:

https://api.example.com:443/users/42?format=json#profile

Its main components are:

The fragment is not sent to the server in an HTTP request. It is interpreted by the user agent, commonly to select a location in an HTML document or application state.

A URL is not the same as a URI in every theoretical usage, but modern web development commonly uses URL terminology for web addresses. Paths should be designed consistently. For example, /users/42 conventionally identifies user 42, while /getUser?id=42 expresses the same operation through a query parameter. Both can work, but predictable resource-oriented paths improve readability and API documentation.

DNS

The Domain Name System (DNS) translates human-readable names into network addresses. When a client requests api.example.com, it may ask a recursive DNS resolver. The resolver can consult authoritative name servers and return an address such as an IPv4 address or IPv6 address.

Important DNS record types include:

• A: maps a name to an IPv4 address.
• AAAA: maps a name to an IPv6 address.
• CNAME: creates an alias to another domain name.
• MX: identifies mail servers.
• TXT: stores text used for verification and policy mechanisms.
• NS: identifies authoritative name servers.

DNS results are cached for a period determined by the record's time to live (TTL). Caching improves performance but means that a change may not be visible immediately to every client. A DNS failure occurs before HTTP begins; therefore, an HTTP status code cannot be expected when the hostname cannot be resolved.

2. The HTTP Request–Response Model

HTTP is an application-layer protocol for transferring representations of resources. A client sends a request, and a server returns a response. A request generally contains a method, target path, headers, and optionally a message body.

Example request:

GET /products/18 HTTP/1.1
Host: shop.example
Accept: application/json
Authorization: Bearer token-value

Example response:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=60

{"id":18,"name":"Keyboard"}

HTTP/1.1 requires a Host header. HTTP/2 and HTTP/3 use more efficient binary framing, but the conceptual model of methods, headers, status codes, and representations remains familiar. HTTP/3 uses QUIC over UDP, whereas traditional HTTP/1.1 and HTTP/2 commonly use TCP. This distinction affects transport performance, not the basic meaning of an HTTP API.

A request body is separate from the URL and headers. It commonly contains JSON, form data, or binary data. The Content-Type header describes the body sent by the client; the Accept header describes representations the client is willing to receive.

3. HTTP Methods and Their Semantics

HTTP methods communicate the intended operation. They are not merely labels for application functions.

A safe method is intended for retrieval and should not request a state-changing action. A safe method may still cause logging, billing for infrastructure, or other incidental effects; “safe” refers to the requested semantics.

An idempotent method produces the same intended server state when the same request is repeated, although responses may differ. For example, deleting an already deleted resource may return 404 on a later attempt, but the intended final state remains “resource absent.” PUT is useful for retries because the client supplies the target representation or complete replacement.

POST is commonly used when the server assigns a new identifier. A successful creation often returns 201 Created and a Location header identifying the new resource.

4. Status Codes and Error Interpretation

HTTP status codes are grouped by their first digit:

401 Unauthorized usually means that authentication is absent or invalid; it does not mean that the user has been authenticated but denied permission. 403 Forbidden generally indicates that the server understood the request but refuses to authorize it.

A useful error response should be machine-readable and safe. For example:

{
  "error": "validation_failed",
  "message": "The request contains invalid fields",
  "fields": {
    "email": "Must be a valid email address"
  }
}

Error messages must not expose passwords, secret keys, database queries, stack traces, or personal information. Applications should log diagnostic details securely on the server while returning an appropriate public response.

5. Headers and Representations

Headers are metadata represented as name–value fields. Common request headers include Host, Authorization, Accept, User-Agent, Origin, If-None-Match, and Content-Type. Common response headers include Content-Type, Content-Length, Location, Set-Cookie, Cache-Control, ETag, Allow, and security headers.

A representation has a media type, often called a MIME type. Examples include application/json, text/html, text/css, and image/png. The server should set Content-Type accurately. A client should not assume that a response is JSON merely because the URL contains /api.

Content Negotiation

Content negotiation allows a client and server to select an appropriate representation. A client might send:

Accept: application/json, text/plain;q=0.8

The quality value q expresses preference, not an absolute guarantee. If the server cannot produce an acceptable representation, it may return 406 Not Acceptable. When a request body is in an unsupported format, the server may return 415 Unsupported Media Type.

Language negotiation can use headers such as:

Accept-Language: en-GB, en;q=0.9

If a response varies according to a request header, the server should identify that header with Vary, for example Vary: Accept-Language. This helps caches avoid serving one representation to an incompatible request.

6. Cookies, Sessions, and Stateless Communication

HTTP is fundamentally stateless: each request contains the information needed for the server to interpret it, and the protocol does not require the server to remember previous requests. Statelessness supports horizontal scaling because any server instance can process a request when the necessary authentication and data are available.

Applications often need continuity, such as login state or a shopping cart. Cookies provide one mechanism. A server sends:

Set-Cookie: session_id=abc123; Secure; HttpOnly; SameSite=Lax; Path=/

The browser later sends:

Cookie: session_id=abc123

The cookie value should normally be an unpredictable identifier rather than sensitive information. The server stores session data in a database, cache, or other shared store. This is called a server-side session.

Important cookie attributes include:

• Secure: sends the cookie only over HTTPS.
• HttpOnly: prevents ordinary JavaScript from reading the cookie.
• SameSite: limits cross-site cookie transmission and helps reduce cross-site request forgery (CSRF).
• Expires or Max-Age: controls lifetime.
• Domain and Path: limit where the cookie is sent.

Cookies do not make HTTP itself stateful. They add an application-level identifier that allows the server to associate separate requests with stored state. An alternative is a signed or encrypted token, such as a JSON Web Token (JWT), sent in an Authorization header. Tokens can reduce server-side session storage but introduce challenges involving expiration, revocation, secure storage, and token leakage.

Stateless design does not mean “store nothing.” It means that the protocol request can be processed without relying on hidden connection history. A service may use a database, cache, or token while preserving stateless request handling.

7. HTTPS and TLS

HTTPS is HTTP carried through a Transport Layer Security (TLS) connection. TLS provides:

1. Confidentiality: outsiders should not read the protected traffic.
2. Integrity: alteration of traffic should be detected.
3. Server authentication: a certificate helps the client verify the server's domain identity.

During the TLS handshake, the client and server negotiate cryptographic parameters, authenticate the server certificate, and establish session keys. Certificate authorities sign certificates that bind domain names to public keys. Developers should use valid certificates, redirect HTTP to HTTPS where appropriate, protect private keys, and avoid disabling certificate verification in production.

HTTPS does not automatically make an application secure. It cannot prevent SQL injection, broken access control, insecure password storage, malicious input, or vulnerabilities in server code. It protects communication in transit, not every component that handles the data.

Security-related headers can strengthen deployments. Strict-Transport-Security instructs compatible browsers to use HTTPS for a specified period. Content-Security-Policy can reduce some script injection risks. These headers require careful configuration and testing because an incorrect policy may break legitimate application behaviour.

8. Cross-Origin Resource Sharing

An origin is defined by scheme, host, and port. For example, https://app.example.com and https://api.example.com have different origins because their hosts differ. Browsers enforce the same-origin policy, which restricts scripts from freely reading responses from another origin.

Cross-Origin Resource Sharing (CORS) is a browser-controlled mechanism that permits a server to declare which cross-origin requests are allowed. A simple request may include an Origin header, and the server may respond with:

Access-Control-Allow-Origin: https://app.example.com

For a request involving a non-simple method or headers, the browser may first send an OPTIONS preflight request:

OPTIONS /payments HTTP/1.1
Origin: https://app.example.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: Authorization, Content-Type

The server can approve it with headers such as Access-Control-Allow-Methods, Access-Control-Allow-Headers, and Access-Control-Max-Age.

CORS is enforced by browsers. It is not an authentication system and does not prevent a non-browser client from sending a request. When credentials such as cookies are used, the server must explicitly permit credentials and cannot use a wildcard * for Access-Control-Allow-Origin. CORS configuration should allow only required origins, methods, and headers.

9. Practical Request Flow and Troubleshooting

A request commonly follows this sequence:

1. The client parses the URL.
2. DNS resolves the hostname.
3. The client establishes a transport connection.
4. For HTTPS, TLS negotiates encryption and certificate validation occurs.
5. The client sends the HTTP request.
6. A proxy, load balancer, or web server routes the request.
7. Application code validates, authenticates, and processes it.
8. The server returns a response.
9. The client interprets status, headers, and body.

Troubleshooting should follow the same layers. Useful commands include:

nslookup api.example.com
curl -i https://api.example.com/health
curl -i -X POST https://api.example.com/users \
  -H 'Content-Type: application/json' \
  -d '{"name":"Amina"}'

nslookup tests name resolution. curl -i displays response headers and body. Browser developer tools display request URLs, methods, headers, timing, cookies, preflight requests, and response data.

A disciplined diagnostic process asks:

• Does DNS resolve the expected address?
• Is the server reachable on the expected port?
• Does the TLS certificate match the hostname and remain valid?
• Is the request method and path correct?
• Are authentication and content-type headers present?
• Is the response status generated by the application or an intermediary?
• Is the browser blocking the response because of CORS?
• Are cookies being excluded because of Secure, SameSite, domain, or path rules?

Applied Case: A Cross-Origin Order Service

Background

This illustrative case concerns a retail application. The user interface is hosted at https://shop.example, while the order API is hosted at https://api.example. The interface can display products, but order submission fails in the browser.

Problem

The browser sends a POST request to /orders, but the developer observes a CORS error. A direct curl request appears to work.

Analysis

The browser sends an OPTIONS preflight because the request uses POST, JSON, and an authorization header. The API responds without Access-Control-Allow-Origin and does not list Authorization in Access-Control-Allow-Headers. The browser therefore prevents the frontend script from reading the response. The API may have processed no order because the preflight was rejected.

The team also discovers that the session cookie lacks Secure and has an unsuitable SameSite setting. Consequently, authenticated browser requests do not consistently include the cookie.

Approach and Solution

The API is configured to allow exactly https://shop.example, the POST method, and the required Content-Type and Authorization headers. The response includes the appropriate CORS headers for approved requests. If cookies are required, credentials are explicitly enabled and the frontend sends requests with credentials enabled. Cookies are marked Secure, HttpOnly, and given a deliberate SameSite policy. The API validates the authenticated user and prevents duplicate orders using an idempotency strategy at the application level.

Outcome

The preflight succeeds, the browser permits the actual request, and authenticated orders are created. Logs distinguish rejected preflight requests, authentication failures, validation errors, and database failures.

Learning Points

CORS is a browser policy, not a substitute for authorization. curl may succeed because it does not enforce browser same-origin restrictions. Cookies, origins, preflight requests, and API authorization must be considered together. Operationally useful logs and clear status codes reduce diagnosis time.

Design Practices and Limitations

Good HTTP services use resource-oriented URLs, correct methods, explicit media types, meaningful status codes, validation, bounded request sizes, timeouts, and consistent error formats. They document authentication requirements and avoid leaking implementation details.

Retries require care. Clients should generally retry transient failures such as some 503 responses, but repeated POST requests can create duplicates unless the operation is designed to be idempotent. Rate limiting, represented commonly by 429 Too Many Requests, should be accompanied by clear client guidance, sometimes through Retry-After.

Caching can improve performance but may expose stale or sensitive data if configured carelessly. Private responses should not be stored in shared caches without appropriate controls. Sensitive values should not be placed in URLs because URLs may appear in browser history, logs, analytics systems, and referrer data.

Privacy and governance are also relevant. Cookies and identifiers may be personal data under applicable law. Organisations should collect only necessary information, protect it in transit and at rest, define retention periods, control access, and provide transparent notices where required. Security testing should include authorization, session management, CORS, TLS configuration, input validation, and error handling.

Chapter Summary

HTTP is an application-layer request–response protocol used to exchange representations of resources. URLs identify resources through schemes, hosts, ports, paths, queries, and optional fragments. DNS translates domain names into network addresses before HTTP communication begins.

HTTP methods express operation semantics. GET is safe and idempotent; POST commonly creates or processes data; PUT replaces a known resource; PATCH partially modifies; DELETE removes; and OPTIONS supports capability discovery and CORS preflight. Status codes communicate outcomes and should be interpreted together with headers and response bodies.

Headers describe requests and responses, while content negotiation uses Accept, Content-Type, and related fields to select representations. HTTP is stateless by design, but cookies, server-side sessions, and tokens provide application-level continuity. Secure cookie attributes and careful session management are essential.

HTTPS uses TLS to provide confidentiality, integrity, and server authentication, but it does not eliminate application vulnerabilities. CORS controls whether browser scripts may read cross-origin responses; it is not an authentication mechanism. Effective troubleshooting follows the request through DNS, transport, TLS, HTTP routing, application logic, and browser policy.

Key Terms

URL: An address identifying a resource and the method used to access it.

DNS: The distributed naming system that maps domain names to network addresses.

HTTP: An application-layer protocol for transferring web resources.

Request: A client message containing a method, target, headers, and optional body.

Response: A server message containing a status, headers, and optional body.

Idempotent: A property whereby repeating an operation has the same intended effect on server state.

Safe method: A method intended for retrieval rather than requested state change.

Status code: A numeric HTTP result indicating informational, successful, redirect, client-error, or server-error processing.

Header: Metadata field describing an HTTP request or response.

Representation: A transferable form of a resource, such as JSON or HTML.

Media type: A label describing the format of content, such as application/json.

Content negotiation: Selection of a suitable representation based on client and server capabilities.

Cookie: Browser-stored data sent with matching requests.

Session: Application-managed continuity associating requests with a user or client.

Statelessness: The property that each request can be interpreted without required hidden protocol history.

HTTPS: HTTP protected by Transport Layer Security.

TLS: A protocol providing encrypted and authenticated communication.

CORS: A browser-controlled mechanism for permitting selected cross-origin requests.

Preflight: An HTTP OPTIONS request used by a browser to ask whether a cross-origin request is permitted.

Origin: The combination of scheme, host, and port used to define a browser security boundary.

Review Questions

Short Answer Questions

1. What are the main components of a URL?
2. Why can a DNS failure occur without an HTTP status code?
3. Distinguish 401 Unauthorized from 403 Forbidden.
4. What is the purpose of the Content-Type header?
5. Why is the HttpOnly cookie attribute useful?

Long Answer Questions

1. Explain the HTTP request–response model and the role of methods, headers, status codes, and bodies.
2. Compare GET, POST, PUT, PATCH, and DELETE, including their safety and idempotency characteristics.
3. Explain how cookies and server-side sessions provide continuity while HTTP remains stateless.
4. Describe the TLS handshake at a conceptual level and explain what HTTPS does and does not protect.
5. Explain CORS, including preflight requests, allowed origins, credentials, and common configuration errors.

Analytical / Application Questions

1. An API returns 415 Unsupported Media Type when a client submits JSON. Identify likely causes and propose a diagnostic sequence.
2. A browser can call an API successfully with curl, but frontend JavaScript reports a CORS error. Explain why the results differ and recommend a server-side solution.
3. Design an order-creation endpoint that remains safe when a client retries after a network timeout. Specify the method, status codes, authentication approach, and strategy for preventing duplicate orders.

Multiple Choice Questions with Answers

Question 1

Which part of a URL is not sent to the server in an HTTP request?

A. The scheme

B. The host

C. The fragment

D. The path

Correct Answer: C. The fragment

Question 2

Which DNS record maps a domain name to an IPv4 address?

A. AAAA

B. A

C. MX

D. CNAME

Correct Answer: B. A

Question 3

Which HTTP method is normally safe and idempotent for retrieving a resource?

A. POST

B. PATCH

C. GET

D. CONNECT

Correct Answer: C. GET

Question 4

Which status code commonly indicates that a new resource was successfully created?

A. 200

B. 201

C. 204

D. 304

Correct Answer: B. 201

Question 5

What does the Content-Type header describe?

A. The server's IP address

B. The format of the message body

C. The user's permissions

D. The connection timeout

Correct Answer: B. The format of the message body

Question 6

Which cookie attribute prevents ordinary JavaScript from reading the cookie?

A. Secure

B. Path

C. HttpOnly

D. Max-Age

Correct Answer: C. HttpOnly

Question 7

What is the primary purpose of HTTPS?

A. To replace DNS

B. To encrypt and authenticate HTTP communication

C. To guarantee valid application code

D. To disable browser security policies

Correct Answer: B. To encrypt and authenticate HTTP communication

Question 8

Why might a browser send an OPTIONS request before a cross-origin POST request?

A. To perform DNS resolution

B. To request a CORS preflight permission

C. To renew a TLS certificate

D. To delete an old cookie

Correct Answer: B. To request a CORS preflight permission

Question 9

Which response most commonly indicates that authentication credentials are missing or invalid?

A. 401 Unauthorized

B. 403 Forbidden

C. 404 Not Found

D. 502 Bad Gateway

Correct Answer: A. 401 Unauthorized

Question 10

Which statement best describes HTTP statelessness?

A. Servers cannot store any data

B. Every request must use a different domain

C. Each request can be interpreted without required hidden protocol history

D. Clients cannot use cookies

Correct Answer: C. Each request can be interpreted without required hidden protocol history