5.2 Delivery semantics
End-to-end exactly-once execution across a broker, database, and external service is generally not obtained merely by selecting a broker option. Each boundary must participate in the guarantee. 6. Idempotency and Safe Payment ProcessingAn operation is idempotent when repeating it with the same logical input does not create additional effects. Setting an order status to A dangerous payment task performs a provider charge and then records success locally. If the worker crashes between those actions, the broker retries the task and may charge the customer twice. A safe design uses a durable, unique operation record and a provider-supported idempotency key: from celery import shared_task
from django.db import transaction
@shared_task(bind=True, autoretry_for=(TemporaryProviderError,),
retry_backoff=True, max_retries=6, acks_late=True)
def charge_order(self, operation_id):
operation = PaymentOperation.objects.get(id=operation_id)
if operation.status == "succeeded":
return operation.provider_charge_id
# The provider durably associates this key with one logical charge.
response = provider_charge(
amount=operation.amount,
currency=operation.currency,
idempotency_key=str(operation.idempotency_key),
)
with transaction.atomic():
locked = PaymentOperation.objects.select_for_update().get(
id=operation_id
)
if locked.status != "succeeded":
locked.status = "succeeded"
locked.provider_charge_id = response.charge_id
locked.save(update_fields=["status", "provider_charge_id"])
return response.charge_idThe API must create If the worker crashes after Holding a database lock during the remote call is usually undesirable because it creates long transactions without making the remote side effect atomic. If a provider offers no idempotency facility, a local database transaction alone cannot guarantee one remote charge. Alternatives include provider-side lookup, a controlled payment gateway, compensating refunds, and manual reconciliation. 7. Transactional Outbox and Event-Driven PatternsA dual-write problem occurs when an application commits a database change and publishes an event as separate actions. A crash between them can leave the database updated without an event, or an event published without the intended state. The transactional outbox pattern writes the business change and an outbox record in one database transaction. A relay later publishes unpublished outbox records. Consumers must still be idempotent because the relay can publish a record more than once. Important event-driven patterns include:
Events should be named as completed facts, such as 8. Capacity, Queueing, and BackpressureSuppose jobs arrive at average rate \lambda jobs per unit time. There are c workers, and each worker completes an average of \mu jobs per the same unit time. A basic traffic-intensity measure is: ρ = λ ÷ (cμ) Here, \rho is dimensionless. The model assumes comparable workers and a work-conserving queue. A necessary stability condition for this basic capacity model is \lambda < c\mu, although that condition alone does not guarantee acceptable latency or cover bursty arrivals, dependencies, or failures. Worked capacity exampleA media service receives 120 jobs per minute. Average service time is 30 seconds, so one worker completes 2 jobs per minute. With 80 workers: ρ = 120 ÷ (80 × 2) = 0.75 Average offered load is therefore 75% of nominal capacity. Since 120 ÷ 2 = 60, more than 60 workers are required for strict average-rate stability; 61 is the smallest integer satisfying the inequality. In practice, additional headroom is required for bursts, retries, uneven job sizes, and worker failures. For a stable system in steady state, Little's Law relates average number in the system L, average arrival rate \lambda, and average time in the system W: L = λW Units must be consistent. If 2 jobs arrive per second and average total time in the system is 8 seconds, then L = 2 × 8 = 16 jobs are present on average. “Time in the system” includes both queue waiting and service time. Backpressure prevents producers from overwhelming consumers. Techniques include bounded queues, rate limits, admission control, consumer flow control, load shedding, and adaptive concurrency limits. 9. Retries and Failure HandlingRetries are appropriate for transient failures such as temporary network loss, rate limiting, or short service outages. They are inappropriate for deterministic validation failures, invalid credentials, or malformed data unless something relevant changes. A capped exponential retry schedule can be represented as: dk = min(dmax, d0 × 2k) + Jk where dk is the delay before retry number k, d0 is the base delay, dmax is the cap, and Jk is a non-negative random jitter selected according to the system's policy. Here k = 0 denotes the first retry. All delay terms must use the same time unit. For d0 = 1 second, a 20-second cap, and zero jitter, delays are 1, 2, 4, 8, 16, 20, 20 seconds. Jitter reduces synchronized retry storms. A complete failure policy should define:
A circuit breaker stops sending requests after repeated failures, waits for a recovery interval, and then allows limited test requests. It protects both the caller and a struggling dependency. Bulkheads isolate resource pools so one failing dependency does not consume every connection or worker. 10. Scheduling Periodic WorkSchedulers initiate work at a future time or on a recurring timetable. Examples include Celery Beat, cron, Kubernetes CronJobs, cloud schedulers, and BullMQ repeatable jobs. Scheduled tasks may overlap when one execution takes longer than its interval. Designers must choose among allowing overlap, applying a distributed lock, skipping the next run, or coalescing missed runs. Time zones and daylight-saving transitions make local-time schedules hazardous. Storage in UTC and an explicit business-time-zone policy are recommended. A scheduler should ordinarily enqueue a durable task rather than perform extensive work itself. In a replicated deployment, leader election or a singleton scheduler prevents unintended duplicate scheduling, while task idempotency provides an additional safeguard. 11. WebSockets and Real-Time DeliveryHTTP request–response communication is client initiated. A WebSocket begins with an HTTP upgrade handshake and then provides a persistent, full-duplex connection over TCP. Either endpoint can send messages without opening a new HTTP request. import { WebSocketServer } from "ws";
const wss = new WebSocketServer({ port: 8080 });
wss.on("connection", (socket, request) => {
const user = authenticateUpgrade(request);
if (!user) return socket.close(1008, "Unauthorized");
socket.on("message", raw => {
const message = JSON.parse(raw.toString());
validateMessage(message);
handleClientEvent(user, message);
});
socket.send(JSON.stringify({ type: "connected" }));
});Django applications commonly implement WebSockets through ASGI and Django Channels. Channel layers can distribute messages across application instances. 11.1 Production concerns
WebSocket frames are ordered on one functioning TCP connection, but reconnects and distributed processing can cause missing, repeated, or differently ordered application events. Sequence numbers and resumable event identifiers can support recovery. Server-Sent Events are a simpler alternative when communication is primarily server-to-client. Conventional polling remains appropriate for infrequent updates and operational simplicity. 12. Illustrative Case: Real-Time Order FulfilmentBackgroundThis fictional case concerns Northstar Market, an online retailer. Its original API synchronously authorized payment, reserved stock, generated an invoice, and sent confirmation email before responding. ProblemResponses frequently exceeded ten seconds. Email outages blocked checkout, duplicate submissions occasionally created duplicate payment attempts, and customers had no visibility into fulfilment progress. AnalysisPayment authorization was essential before confirming an order, but invoice generation and email delivery did not need to delay the response. Broker redelivery meant workers could receive tasks more than once. Database updates and event publication also formed a dual-write risk. ApproachThe team introduced the following design:
OutcomeIn the case's controlled load test, the API acknowledgement time fell from several seconds to below 300 milliseconds for most requests because nonessential work left the request path. Duplicate broker deliveries did not create duplicate charges because the payment provider enforced the stored idempotency key. Queue-depth alerts exposed an undersized invoice-worker pool before user-facing deadlines were breached. Learning PointsAsynchrony reduced request latency but did not eliminate work. Reliability depended on durable state, provider-supported idempotency, consumer deduplication, reconciliation, observability, and explicit failure policies. WebSockets improved visibility but remained a notification channel rather than the sole source of truth. 13. Observability, Security, and GovernanceAsynchronous systems separate cause and effect in time and across processes. Logs should therefore include correlation IDs, operation IDs, task IDs, attempt numbers, and trace context. Useful metrics include queue depth, oldest-message age, task duration, failure rate, retry count, dead-letter count, event-loop delay, WebSocket connection count, and consumer lag. Security controls must protect both transport and message content. Broker connections require encryption and least-privilege credentials. Sensitive payloads should be minimized because queues, logs, and dead-letter stores may retain data longer than web requests. Retention and deletion policies must comply with privacy obligations. Automated retries can amplify harm. A defective AI inference task, notification process, or financial workflow may repeat an incorrect action at scale. Human review, spending limits, kill switches, audit trails, and model/version metadata are appropriate for high-impact systems. Operational ownership must be explicit. Teams need runbooks for replaying dead-letter messages, reconciling uncertain outcomes, rotating broker credentials, and responding to queue backlogs. Replaying historical events should occur in a controlled environment because current consumers may interpret old schemas differently. 14. Best-Practice Checklist
Chapter SummaryAsynchronous systems improve responsiveness by allowing applications to progress while operations wait. Node.js supports I/O concurrency through its event loop, promises, and Task queues separate slow or retryable work from HTTP requests. Because common systems provide at-least-once delivery, tasks must tolerate duplicate execution. Financial operations require durable local state plus provider-side idempotency; a local status check alone cannot prevent duplicate remote effects after a crash. Transactional outbox, reconciliation, dead-letter handling, exponential backoff, and observability address common distributed failure modes. WebSockets support low-latency bidirectional communication, but production deployments must manage authentication, slow consumers, reconnection, replay, and distributed ordering. Reliable asynchronous design ultimately depends on explicit contracts for durability, idempotency, capacity, failure recovery, and governance. Key TermsAsynchrony: Execution that permits work to continue while an operation awaits completion. Concurrency: Overlapping progress of multiple tasks. Parallelism: Simultaneous execution on multiple processing resources. Promise: A JavaScript object representing an eventual fulfilled or rejected result. Event loop: A runtime mechanism that schedules callbacks and reacts to completed events. ASGI: Python interface supporting asynchronous servers, applications, and long-lived connections. Background job: Work executed outside the immediate request–response path. Message broker: Infrastructure that stores, routes, and delivers messages between components. At-least-once delivery: Delivery model in which retries prevent loss but can produce duplicates. Idempotency: Property whereby repeating an operation has one logical effect. Idempotency key: Stable identifier used to recognize retries of the same logical operation. Transactional outbox: Pattern that atomically stores business changes and events for later publication. Dead-letter queue: Isolation area for messages that cannot be processed successfully. Backpressure: Mechanism that limits producers when consumers cannot keep pace. Exponential backoff: Retry policy in which delays grow exponentially up to a cap. Jitter: Random variation added to retry timing to prevent synchronized retries. Circuit breaker: Control that temporarily blocks calls to a repeatedly failing dependency. WebSocket: Persistent full-duplex communication protocol established through an HTTP upgrade. Reconciliation: Process of comparing systems to resolve uncertain or inconsistent outcomes. Saga: Distributed workflow composed of local transactions and compensating actions. Review QuestionsShort Answer Questions
Long Answer Questions
Analytical / Application Questions
Multiple Choice Questions with AnswersQuestion 1Which statement correctly distinguishes concurrency from parallelism? A. Concurrency is overlapping progress; parallelism is simultaneous execution on multiple processing resources. B. Concurrency requires multiple CPU cores, while parallelism does not. C. Concurrency applies only to I/O, while parallelism applies only to databases. D. The terms describe exactly the same execution behavior. Correct Answer: A. Concurrency is overlapping progress; parallelism is simultaneous execution on multiple processing resources. Question 2Two independent promises must both succeed before processing continues. Which JavaScript construct is most appropriate? A. Promise.race() B. Promise.all() C. Promise.reject() D. setTimeout() Correct Answer: B. Promise.all() Question 3Why can CPU-intensive synchronous JavaScript reduce Node.js server responsiveness? A. It deletes pending promises. B. It disables the operating system scheduler. C. It occupies the event-loop thread and delays other callbacks. D. It converts all network requests into blocking file operations. Correct Answer: C. It occupies the event-loop thread and delays other callbacks. Question 4Which deployment interface enables Django's full asynchronous request stack and long-lived connections? A. CGI B. WSGI only C. ASGI D. SQLAlchemy Correct Answer: C. ASGI Question 5In the chapter's asynchronous API architecture, which direction does the HTTP A. From the worker to the broker B. From the client to the API C. From the database to the worker D. From the API to the client Correct Answer: D. From the API to the client Question 6What most directly prevents a duplicate external payment when a worker crashes after charging but before saving local success? A. A longer database lock held only after the provider call B. A stable provider-supported idempotency key reused by retries C. Returning HTTP 200 instead of HTTP 202 D. Increasing the number of workers Correct Answer: B. A stable provider-supported idempotency key reused by retries Question 7What problem does the transactional outbox pattern primarily address? A. Atomic coordination of a database change and its event-publication intent B. Encryption of WebSocket frames C. Parallel execution of CPU-intensive JavaScript D. Automatic scaling of database indexes Correct Answer: A. Atomic coordination of a database change and its event-publication intent Question 8A queue receives 120 jobs per minute. Each of 80 workers completes 2 jobs per minute on average. What is the traffic intensity \rho = \lambda ÷ (c\mu)? A. 0.50 B. 0.60 C. 0.75 D. 1.50 Correct Answer: C. 0.75 Question 9Using dk = min(dmax, d0 × 2k) with no jitter, d0 = 2 seconds, dmax = 20 seconds, and k = 3, what is the retry delay? A. 6 seconds B. 8 seconds C. 16 seconds D. 20 seconds Correct Answer: C. 16 seconds Question 10Which mechanism is specifically intended to protect a system from repeatedly calling a failing dependency? A. Circuit breaker B. WebSocket heartbeat C. Promise fulfillment D. Database normalization Correct Answer: A. Circuit breaker
Backend Frameworks (Node.js/Django)
|
Chapter 8 of 14 |