APIs, Integration and Asynchronous Work

Introduction

Modern Python applications rarely operate in isolation. A desktop program may obtain weather data from a remote service, a web application may process payments through a third-party provider, and an administrative dashboard may communicate with several internal services. Application Programming Interfaces (APIs) provide the contracts through which these systems exchange data and functionality.

This chapter develops practical foundations for designing and consuming JSON REST APIs with Python. It also examines payload validation, error handling, interface versioning, third-party integration, background jobs, caching, and real-time communication. The emphasis is on maintainable application design rather than merely sending HTTP requests.

The examples use Flask, the requests library, and standard Python techniques. In production, asynchronous frameworks such as FastAPI, ASGI servers, task queues, and WebSocket libraries may also be appropriate.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and constraints of REST-style APIs.
2. Design JSON resources, routes, status codes, and response formats.
3. Consume APIs safely from Python applications.
4. Validate request payloads and produce useful error responses.
5. Handle timeouts, authentication failures, rate limits, and partial failures.
6. Explain API versioning and backward compatibility.
7. Integrate third-party services securely and responsibly.
8. Distinguish synchronous requests, background jobs, caching, polling, Server-Sent Events, and WebSockets.
9. Select an appropriate integration pattern for a practical application.

1. APIs and HTTP Foundations

An API is an interface that defines how one software component requests services from another. A web API commonly uses HTTP, the protocol underlying the World Wide Web. A client sends a request and a server returns a response.

An HTTP request contains a method, a URL, headers, and sometimes a body. Common methods include:

A method is safe when it is intended not to change server state. A method is idempotent when repeating the same request has the same intended effect as making it once. Idempotency does not mean that every response is identical; it concerns the resulting state.

A response contains a status code, headers, and an optional body. Important status-code classes are:

• 2xx success: the request was processed successfully.
• 3xx redirection: the client should use another location or cached representation.
• 4xx client error: the request is invalid, unauthorised, forbidden, or refers to a missing resource.
• 5xx server error: the server could not fulfil an otherwise potentially valid request.

Examples include 200 OK, 201 Created, 204 No Content, 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 Conflict, 422 Unprocessable Content, 429 Too Many Requests, and 503 Service Unavailable.

REST-style Design

Representational State Transfer (REST) is a style of designing networked systems. A REST-style API generally uses identifiable resources, representations such as JSON, standard HTTP methods, stateless requests, and meaningful status codes. REST is not a single library or a mandatory specification; it is a set of architectural constraints and conventions.

For a library application, resource-oriented routes might be:

GET    /api/v1/books
GET    /api/v1/books/42
POST   /api/v1/books
PATCH  /api/v1/books/42
DELETE /api/v1/books/42

A route should express a resource rather than an action wherever possible. /books/42 is generally clearer than /getBook?id=42. Actions that do not map naturally to CRUD can use carefully named subresources, such as /orders/42/cancel.

A useful JSON representation should be consistent and explicit:

{
  "id": 42,
  "title": "Practical Python",
  "available": true,
  "created_at": "2025-02-10T09:30:00Z"
}

Dates should use a documented format, normally an unambiguous ISO 8601 representation in UTC. Clients should not be expected to infer whether a number represents cents, dollars, kilograms, or another unit; the API documentation must specify such meanings.

2. Designing a Flask JSON API

Flask maps HTTP requests to Python functions. A small but structured API can be implemented as follows:

from flask import Flask, jsonify, request

app = Flask(__name__)
books = {}
next_id = 1

@app.get("/api/v1/books/<int:book_id>")
def get_book(book_id):
    book = books.get(book_id)
    if book is None:
        return jsonify({"error": {
            "code": "BOOK_NOT_FOUND",
            "message": "The requested book does not exist"
        }}), 404
    return jsonify(book), 200

@app.post("/api/v1/books")
def create_book():
    global next_id
    data = request.get_json(silent=True)
    if not isinstance(data, dict):
        return jsonify({"error": {
            "code": "INVALID_JSON",
            "message": "A JSON object is required"
        }}), 400

    title = data.get("title")
    if not isinstance(title, str) or not title.strip():
        return jsonify({"error": {
            "code": "INVALID_TITLE",
            "message": "title must be a non-empty string"
        }}), 422

    book = {"id": next_id, "title": title.strip(), "available": True}
    books[next_id] = book
    next_id += 1
    return jsonify(book), 201

The example demonstrates several design principles: the server parses JSON explicitly, validates the top-level type, rejects invalid input, returns machine-readable error codes, and uses 201 Created for successful creation. A real application would use a database, authentication, schema validation, logging, and transaction management.

Resource Collections, Pagination, and Filtering

Returning thousands of records in one response is inefficient. Collection endpoints should support pagination and filtering. A common contract is:

GET /api/v1/books?limit=20&offset=40&available=true

The response can include both data and navigation information:

{
  "items": [
    {"id": 41, "title": "Networks", "available": true}
  ],
  "limit": 20,
  "offset": 40,
  "total": 187
}

The API must define maximum limits, ordering, and whether total is exact. Cursor pagination is often more stable than offset pagination for frequently changing datasets because a cursor identifies a position in an ordered result rather than relying on row numbers.

3. Consuming JSON APIs with Python

The requests library provides a practical synchronous client:

import requests


def fetch_book(book_id, token):
    url = f"https://catalog.example.com/api/v1/books/{book_id}"
    headers = {"Authorization": f"Bearer {token}", "Accept": "application/json"}

    response = requests.get(url, headers=headers, timeout=(3.05, 10))
    response.raise_for_status()
    return response.json()

A timeout is essential. Without one, a network operation can wait indefinitely and consume application resources. The tuple (3.05, 10) specifies a connection timeout and a read timeout in seconds. Timeout values should reflect the service and user experience; they are not universal constants.

raise_for_status() converts 4xx and 5xx responses into exceptions, but robust software usually handles expected cases explicitly:

import requests


def safe_fetch(url):
    try:
        response = requests.get(url, timeout=8)
        if response.status_code == 404:
            return None
        response.raise_for_status()
        return response.json()
    except requests.Timeout:
        raise RuntimeError("The external service timed out")
    except requests.HTTPError as exc:
        raise RuntimeError(f"External service failure: {exc.response.status_code}")
    except ValueError:
        raise RuntimeError("The service returned invalid JSON")

Applications should distinguish transport failures, HTTP failures, malformed data, and business-level errors. A successful HTTP response may still contain an application error field, and a valid JSON document may still violate the expected schema.

4. Payload Validation and Error Contracts

Validation protects both the application and its users. It should occur at trust boundaries: data received from browsers, mobile clients, partner services, files, and environment configuration should not be assumed to be correct.

Validation commonly includes:

• required fields;
• permitted data types;
• string length and format;
• numeric range;
• enumerated values;
• relationships between fields; and
• protection against unexpected fields where strictness is appropriate.

A schema library such as Pydantic, Marshmallow, or jsonschema can centralise these rules. A Pydantic-style model illustrates the idea:

from pydantic import BaseModel, Field

class BookInput(BaseModel):
    title: str = Field(min_length=1, max_length=200)
    year: int = Field(ge=1450, le=2100)

The exact syntax depends on the installed library version. Validation should be treated as a design decision: rejecting unknown fields can expose client mistakes early, while accepting them may support forward compatibility. The choice must be documented.

A consistent error envelope helps clients respond correctly:

{
  "error": {
    "code": "VALIDATION_FAILED",
    "message": "One or more fields are invalid",
    "details": [
      {"field": "year", "code": "OUT_OF_RANGE"}
    ],
    "request_id": "7f1c..."
  }
}

Error messages should be useful but must not disclose passwords, access tokens, database queries, stack traces, or internal infrastructure. A request identifier allows support staff to locate detailed server logs without exposing those logs to the client.

5. Reliability, Security, and Resilience

Authentication and Authorisation

Authentication establishes identity; authorisation determines what that identity may do. Common API mechanisms include API keys, HTTP Basic authentication over TLS, OAuth 2.0 access tokens, and signed tokens. Credentials should be transmitted only over HTTPS and stored in environment variables or a secret-management system rather than source code.

A bearer token should be treated like a password: anyone possessing it may use it until it expires or is revoked. Tokens should have least privilege, limited lifetime, and appropriate audience and scope restrictions.

Retries and Idempotency

Transient failures can justify retries, but retrying every failure is dangerous. Do not blindly retry invalid requests, authentication failures, or non-idempotent operations. Use exponential backoff with jitter when a service permits retries. For an attempt number k, a conceptual delay may be expressed as:

Here, D_k is the delay before attempt k, D_0 is the initial delay, D_max is the maximum delay, and J is a random jitter value. The formula is a design pattern, not a universal standard; actual bounds and retry counts must follow the provider's policy.

For a payment or order-creation request, a client can send an idempotency key. The server stores the result associated with that key and returns the same logical result when the request is repeated. This prevents a timeout from causing duplicate business operations.

Rate Limiting and Circuit Breaking

An API may restrict requests per user, token, or IP address. A 429 Too Many Requests response should be handled according to the Retry-After header when present. A client should avoid creating a retry storm.

A circuit breaker stops calling a failing dependency temporarily. It generally has closed, open, and half-open states. This protects the application and gives the dependency time to recover. Circuit breakers complement, rather than replace, timeouts and sensible retries.

Security Risks

Important controls include input validation, output encoding where relevant, HTTPS, authentication, authorisation, CSRF protection for cookie-authenticated browser requests, rate limiting, dependency updates, audit logging, and protection against sensitive-data leakage. Never trust a client-supplied user ID merely because it appears in a URL; verify that the authenticated user has access to the resource.

6. API Versioning and Evolution

An API is a contract. Changing a field from a number to a string, removing a response property, changing error meanings, or altering authentication requirements can break clients.

Common versioning strategies are:

There is no universally best strategy. A diploma-level project should choose one, document it, and apply it consistently. Backward-compatible additions, such as adding an optional response field, are usually safer than removals or semantic changes. Deprecation should include documentation, warnings, a migration guide, and a defined support period.

Contract tests verify that a provider still meets the promises expected by clients. OpenAPI can document routes, parameters, schemas, authentication, and responses, and can support generated client libraries and automated validation. Documentation is not a substitute for tests, but it makes the contract inspectable.

7. Third-Party Service Integration

Third-party integration is a socio-technical problem, not just an HTTP problem. Before coding, evaluate the provider's authentication model, pricing, quotas, service-level commitments, data-retention policy, regional availability, webhook support, version policy, and failure behaviour.

A sound integration separates provider-specific code from business logic. For example:

class WeatherProvider:
    def __init__(self, http_client, api_key):
        self.http_client = http_client
        self.api_key = api_key

    def current_temperature(self, city):
        response = self.http_client.get(
            "https://weather.example/v1/current",
            params={"city": city},
            headers={"X-API-Key": self.api_key},
            timeout=5,
        )
        response.raise_for_status()
        payload = response.json()
        return float(payload["temperature_c"])

Dependency injection makes the provider replaceable in tests. A fake HTTP client can simulate success, timeout, malformed JSON, or a rate-limit response without contacting the real service.

Webhooks

A webhook allows a provider to notify an application when an event occurs. A webhook endpoint should verify a signature, reject stale or duplicated messages where applicable, acknowledge quickly, and process the event asynchronously. Event identifiers should be stored so that delivery retries do not create duplicate effects.

Never trust a webhook merely because it arrived at a secret-looking URL. Verify its cryptographic signature using the provider's documented procedure and protect the endpoint against replay attacks.

8. Background Jobs and Asynchronous Work

A normal Flask request is synchronous from the user's perspective: the server receives a request, performs work, and returns a response. Long operations such as report generation, video conversion, email delivery, or bulk imports should not occupy a web worker for several minutes.

A background-job architecture typically contains:

1. a web application that validates and enqueues work;
2. a durable broker or queue;
3. one or more workers;
4. a result store or database; and
5. monitoring, retry, and dead-letter handling.

A typical response is 202 Accepted with a job resource:

{
  "job_id": "job-4815",
  "status": "queued",
  "status_url": "/api/v1/jobs/job-4815"
}

The client can poll the status endpoint, receive a webhook, or subscribe to a real-time channel. Common Python tools include Celery with Redis or RabbitMQ, RQ, Dramatiq, and cloud-managed queues. The correct choice depends on durability, deployment, throughput, operational expertise, and cost.

Workers must be designed for duplicate delivery. A job should be idempotent where possible, or should use a unique business key and transaction to prevent repeated effects. Retries need a maximum attempt count and a dead-letter or failed state. A job that always fails should not retry forever.

Concurrency Models

Concurrency is not synonymous with parallelism. Concurrency structures overlapping work; parallelism executes work simultaneously on multiple CPU cores. Python's asyncio is effective for I/O-bound work when libraries support asynchronous operations. Threads can also overlap blocking I/O, while CPU-bound work may require multiprocessing or an external worker system.

An asynchronous function should not call a long blocking operation directly on the event loop. Such calls delay unrelated requests. Use an asynchronous client or move blocking work to a suitable executor or worker.

9. Caching

Caching stores a reusable result so that later requests can be served more quickly or with fewer calls to a dependency. A cache entry has a key, value, and usually a time-to-live (TTL). A cache hit returns stored data; a cache miss computes or retrieves the data and may store it.

Caching is useful for public or slowly changing data, expensive computations, and rate-limited providers. It is risky for rapidly changing or private data unless keys include the correct user and authorisation context.

HTTP caching uses headers such as Cache-Control, ETag, and Last-Modified. With an ETag, a client can ask whether its representation remains current. The server may return 304 Not Modified, avoiding transmission of the full body.

The central difficulty is invalidation. After a book is edited, cached representations of that book and possibly related lists may be stale. A practical policy must specify TTL, invalidation events, maximum object size, and behaviour when the cache is unavailable. The cache should normally be an optimisation, not the only copy of critical data.

10. Real-Time Communication Options

Different requirements call for different communication patterns:

Polling is simple but can waste requests and introduce delay. Server-Sent Events use a long-lived HTTP connection and are naturally one-way. WebSockets allow bidirectional communication but require connection management, authentication, reconnection logic, message validation, and scaling strategies such as a shared pub/sub layer.

Real-time delivery does not eliminate the need for durable state. A client that disconnects must be able to resynchronise through an ordinary API or event history. Messages should have identifiers or sequence numbers where ordering and recovery matter.

11. Applied Case: A Library Borrowing Platform

Background

This illustrative case concerns a college library that wants a web and Tkinter client for searching books, borrowing items, and receiving notifications when reserved books become available.

Problem

The catalogue contains 80,000 records, the external notification provider limits requests, and book-availability updates can take several seconds. The original prototype performs every operation synchronously in the browser and occasionally creates duplicate reservations when users retry after a timeout.

Analysis

The system needs a resource-oriented API, pagination, validation, authentication, idempotent reservation creation, caching for catalogue searches, and asynchronous notification delivery. Availability changes are events that may be delivered through a webhook or internal message queue.

Approach and Solution

The API exposes /api/v1/books, /api/v1/reservations, and /api/v1/jobs. Search requests use cursor pagination and a short cache TTL. A reservation request includes an idempotency key. The server validates the authenticated user's permission, checks availability within a database transaction, records the key and result, and returns 201 Created or an appropriate conflict response.

Email notifications are placed on a durable queue. A worker sends the email and records the provider's message identifier. Failed deliveries retry with bounded backoff; permanently failed messages enter a dead-letter queue for investigation.

A Tkinter client uses a worker thread or asynchronous task for network calls so that the graphical user interface remains responsive. The client displays clear states such as “loading”, “reserved”, “already reserved”, and “service temporarily unavailable”. It never displays an access token in a log or error dialog.

Availability notifications use Server-Sent Events for the web client. The client still calls the normal API after reconnecting to obtain the current authoritative state, because a real-time message may have been missed.

Outcome

The revised architecture reduces duplicate reservations, prevents long external calls from blocking web workers, limits catalogue traffic, and gives users understandable feedback. Operational staff can trace a request using a request ID and inspect failed background jobs without exposing personal data.

Learning Points

The case demonstrates that integration design includes data contracts, user experience, security, failure recovery, and operations. A queue solves latency problems but introduces delivery and monitoring responsibilities. Caching improves efficiency but requires an explicit freshness policy. Real-time communication supplements, rather than replaces, a reliable API.

12. Testing, Observability, and Governance

API tests should cover successful requests, malformed JSON, missing fields, invalid types, unauthorised access, resource absence, conflicts, pagination boundaries, rate limits, and dependency failures. Integration tests verify real database and provider behaviour in a controlled environment; contract tests verify the interface between independently deployed services.

Observability combines logs, metrics, and traces. Useful metrics include request count, latency percentiles, error rate, queue depth, job age, cache hit ratio, and external-provider failure rate. Logs should include correlation or request IDs, but should exclude tokens, passwords, unnecessary personal information, and full payment data.

Governance includes data minimisation, retention limits, access controls, audit trails, consent where required, and compliance with applicable privacy law. A third-party provider may transfer data across borders or retain it for its own purposes. Teams should assess these consequences before integration, not after deployment.

AI services deserve the same controls. Prompts and returned content may contain confidential information, generated output may be inaccurate, and provider retention policies may differ. Validate structured AI output, apply access controls, disclose material automated decisions where required, and retain human review for consequential decisions.

Best-Practice Checklist

• Design resources and use HTTP methods and status codes consistently.
• Validate every external payload at the trust boundary.
• Publish schemas, examples, authentication rules, and error contracts.
• Set connection and read timeouts for outbound calls.
• Retry only when safe and use bounded backoff.
• Use idempotency keys for retry-sensitive commands.
• Keep provider-specific code behind an adapter or service boundary.
• Move long-running work to durable background jobs.
• Make workers tolerant of duplicate delivery.
• Define cache freshness and invalidation rules.
• Protect secrets and avoid sensitive data in logs.
• Test failure paths, not only successful paths.
• Monitor latency, errors, queues, dependencies, and cost.
• Plan versioning and deprecation before breaking clients.

Chapter Summary

APIs are contracts through which software components exchange data and services. A well-designed JSON REST API uses resource-oriented routes, appropriate HTTP methods, meaningful status codes, consistent representations, pagination, and documented errors. Flask can implement such interfaces, while Python clients such as requests can consume them safely when they use timeouts and explicit failure handling.

Validation is essential at every trust boundary. Authentication, authorisation, HTTPS, rate limiting, careful logging, and secret management protect the interface. Reliability requires distinguishing transient failures from permanent errors, using bounded retries, idempotency keys, circuit breakers, and durable job processing.

Versioning preserves compatibility as APIs evolve. Third-party integrations should be isolated behind adapters and evaluated for security, privacy, availability, cost, and provider policy. Background queues are appropriate for long-running work, whereas caching reduces repeated computation or network traffic when freshness rules are clear. Polling, Server-Sent Events, WebSockets, and webhooks provide different real-time capabilities and operational costs. Effective integration is therefore a combination of programming, architecture, testing, governance, and user-centred failure design.

Key Terms

API: A defined interface through which software components communicate.

REST: An architectural style using resources, representations, stateless requests, and standard web semantics.

JSON: A text-based data-interchange format based on objects, arrays, strings, numbers, booleans, and null.

Idempotent: Having the same intended state-changing effect when repeated.

Payload: The data carried in a request or response body.

Schema: A formal description of permitted data structure and constraints.

Status code: A numerical HTTP response classification indicating result or failure.

Authentication: Verification of a user's or system's identity.

Authorisation: Determination of what an authenticated identity may do.

Timeout: A limit on how long a network operation may wait.

Exponential backoff: A retry strategy in which delays increase between attempts.

Idempotency key: A unique client-supplied key allowing a server to deduplicate repeated commands.

Webhook: An HTTP callback used to notify another system about an event.

Background job: Work executed outside the immediate request-response cycle.

Message queue: A system that stores work or events until consumers process them.

Cache: A temporary store of reusable data or representations.

TTL: Time to live; the period before cached data expires.

WebSocket: A persistent, bidirectional communication channel over a network connection.

Circuit breaker: A resilience mechanism that temporarily stops calls to a failing dependency.

Contract test: A test verifying that a service meets an agreed interface contract.

Review Questions

Short Answer Questions

1. What is the difference between authentication and authorisation?
2. Why should an outbound HTTP request have a timeout?
3. State two advantages of using a consistent JSON error envelope.
4. What is the purpose of an idempotency key?
5. Name two situations in which a background job is preferable to direct request processing.

Long Answer Questions

1. Explain how REST-style resource design, HTTP methods, and status codes work together in a library API.
2. Describe a robust strategy for validating and reporting errors in a JSON API.
3. Compare URL-path, query-parameter, and header-based API versioning.
4. Explain the architecture and reliability requirements of a background-job system.
5. Compare polling, Server-Sent Events, WebSockets, and webhooks, including a suitable use case for each.

Analytical / Application Questions

1. A payment request times out after the provider may have completed the transaction. Design a safe retry and deduplication strategy.
2. A dashboard must show job progress while supporting mobile clients with intermittent connectivity. Select an appropriate communication design and justify how clients resynchronise state.
3. An external catalogue API begins returning 429 responses and charging per request. Propose changes involving caching, rate-limit handling, observability, and user experience.

Multiple Choice Questions with Answers

Question 1

Which HTTP status code most appropriately indicates that a new resource was successfully created?

A. 200 OK

B. 201 Created

C. 204 No Content

D. 304 Not Modified

Correct Answer: B. 201 Created

Question 2

What is the primary purpose of an outbound network timeout?

A. To encrypt the response

B. To prevent indefinite waiting for the dependency

C. To validate JSON field types

D. To version the API

Correct Answer: B. To prevent indefinite waiting for the dependency

Question 3

Which practice best prevents duplicate effects when a client retries an order-creation request?

A. Removing authentication

B. Using an idempotency key

C. Increasing the response body size

D. Disabling status codes

Correct Answer: B. Using an idempotency key

Question 4

Which component is most appropriate for processing a long-running report without keeping a web request open?

A. A background-job worker

B. A CSS stylesheet

C. A URL fragment

D. A DNS record

Correct Answer: A. A background-job worker

Question 5

What does a 429 HTTP response normally indicate?

A. The resource was permanently deleted

B. The client is being rate limited

C. The server returned valid cached data

D. The request created a resource

Correct Answer: B. The client is being rate limited

Question 6

Which communication method provides a persistent bidirectional channel?

A. Polling

B. Server-Sent Events

C. WebSockets

D. A static HTML file

Correct Answer: C. WebSockets

Question 7

Why should API payloads be validated at a trust boundary?

A. All clients always send correct data

B. External data may violate expected types, constraints, or security assumptions

C. Validation eliminates the need for authentication

D. Validation guarantees network availability

Correct Answer: B. External data may violate expected types, constraints, or security assumptions

Question 8

What is a principal risk of caching private data without including user context in the cache key?

A. The data becomes compressed

B. One user may receive another user's data

C. The API automatically becomes versioned

D. The server cannot use HTTPS

Correct Answer: B. One user may receive another user's data

Question 9

Which mechanism lets a provider notify an application when an event occurs?

A. Webhook

B. Local variable

C. SQL comment

D. Browser bookmark

Correct Answer: A. Webhook

Question 10

What is the main purpose of API versioning?

A. To make every request asynchronous

B. To preserve compatibility while interfaces evolve

C. To replace input validation

D. To remove the need for documentation

Correct Answer: B. To preserve compatibility while interfaces evolve