10. Distributed Systems, Scalability and Resilience

Introduction and Context

A distributed system is a collection of independent computing components that cooperate over a network to provide a service that appears, from the user’s perspective, to be a single application. Modern full-stack systems are distributed almost by necessity. A browser may communicate with an API gateway, authentication service, application services, databases, caches, message brokers, object stores, and external payment or identity providers. These components may run in different processes, containers, availability zones, or geographic regions.

Distribution creates opportunities and difficulties. It enables horizontal scaling, independent deployment, geographic reach, and fault isolation. At the same time, a network call is slower and less reliable than a local function call. Messages may be delayed, duplicated, reordered, or lost. One service may fail while others remain available. Data replicas may temporarily disagree. Consequently, distributed architecture is not simply the decomposition of a large application into microservices; it is the disciplined management of uncertainty, coordination, and failure.

This chapter examines the architectural decisions required to build systems that remain responsive and useful as demand increases and components fail. It focuses on latency, partial failure, consistency, availability, CAP-related trade-offs, load balancing, replication, partitioning, caching, autoscaling, fault isolation, graceful degradation, disaster recovery, and performance engineering.

Learning Objectives

By the end of this chapter, a learner should be able to:

• explain why distributed systems experience latency, inconsistency, and partial failure;
• distinguish scalability, availability, reliability, resilience, and performance;
• apply CAP-related reasoning to database and service architecture;
• select appropriate load-balancing, replication, partitioning, and caching strategies;
• design fault isolation, graceful degradation, retry, timeout, and circuit-breaker mechanisms;
• define meaningful service-level objectives and recovery targets;
• analyse a distributed architecture using capacity, failure, and performance models; and
• evaluate contemporary approaches such as multi-region deployment, event-driven design, and intelligent autoscaling.

The Distributed-System Problem

In a monolithic local program, a function call normally either returns or raises an error within a predictable execution environment. In a distributed system, communication crosses a network boundary. A request can succeed at the server while the response is lost, leaving the client uncertain about the outcome. A server can be healthy but unreachable from one network segment. A clock can differ between machines. A process can pause because of garbage collection or resource exhaustion. These conditions produce what is known as partial failure: only part of the system has failed, while the remainder continues operating.

The most important consequence is uncertainty. A timeout does not prove that an operation was not performed. A successful write to one replica does not guarantee that every reader can immediately see it. A retry may recover from a transient fault, but may also duplicate a non-idempotent operation. Distributed design therefore requires explicit policies for time, identity, ordering, duplication, and recovery.

Latency and the Cost of Coordination

Latency is the elapsed time between an action and its observable result. It includes network transmission, queueing, connection establishment, serialization, server processing, storage access, and contention. Average latency is insufficient for architecture decisions because a small number of very slow requests can affect users and cause cascading timeouts. Percentiles such as p95, p99, and p99.9 are more informative.

If a request performs several independent remote calls in parallel, its latency is often close to the slowest call plus coordination overhead. If calls are sequential, their latencies accumulate. A page requiring six sequential service calls is more vulnerable to latency than one using a single backend composition call. Reducing unnecessary network hops, placing services near their data, reusing connections, batching operations, and parallelising independent work are common latency strategies.

The end-to-end principle also matters: a lower layer cannot guarantee an application-level outcome that only the application understands. Transport-level delivery does not guarantee business-level completion. Payment, order creation, and inventory reservation therefore need application-level identifiers and reconciliation processes.

Availability, Reliability, Resilience, and Scalability

Availability is the proportion of time a service is usable. Reliability concerns the probability that the service performs correctly over a specified period. Resilience is the ability to withstand disturbance and recover while maintaining acceptable service. Scalability is the ability to handle increased workload by adding resources or changing capacity.

These properties are related but not interchangeable. A service may be highly available but return incorrect data, making it unreliable. It may be reliable under normal conditions but non-resilient during a regional outage. A system may scale throughput successfully while its tail latency becomes unacceptable.

A useful availability approximation for independent components in series is:

`A_total = A_1 × A_2 × ... × A_n`

Thus, adding synchronous dependencies can reduce the availability of a user operation. Redundancy and parallel alternatives can improve it, although correlated failures must be considered.

CAP-Related Reasoning and Consistency Choices

The CAP theorem states that when a network partition prevents some nodes from communicating, a distributed data system cannot simultaneously guarantee both strong consistency and availability for all requests. Partition tolerance is not normally optional in a networked system: communication failures occur. The practical decision is therefore how the system behaves during a partition.

A consistency-oriented design may reject or delay operations until it can establish an authoritative ordering or quorum. This protects invariants but reduces availability. An availability-oriented design may accept operations on reachable nodes and reconcile them later. This improves responsiveness but requires conflict resolution and exposes clients to stale or divergent views.

CAP should not be misused as a simple label for an entire database. Different operations may have different requirements. An account-balance update may require linearizable or serializable behaviour, while product recommendations can tolerate stale data. Many systems offer tunable consistency: a read may require one replica, a quorum, or the leader; a write may require acknowledgement from one or several replicas.

Consistency models form a spectrum. Strong consistency makes an operation appear to take effect at one point in a global order. Sequential consistency preserves the order of operations for each process but not necessarily one universal real-time order. Eventual consistency guarantees that replicas converge if updates stop, without guaranteeing immediate agreement. Causal consistency preserves cause-and-effect relationships. The appropriate model depends on business invariants, user expectations, and operational cost.

Architectural Techniques for Scale

Load Balancing

Load balancing distributes requests across service instances. Layer 4 balancing uses transport information such as addresses and ports; layer 7 balancing can use HTTP paths, headers, cookies, or content-aware rules. Common algorithms include round robin, weighted routing, least connections, and latency-aware routing.

A robust design uses health checks that test meaningful readiness rather than merely process existence. It distinguishes a live instance from a ready instance whose dependencies and capacity are adequate. Connection draining allows existing requests to finish before shutdown. Session affinity can simplify stateful applications but reduces balancing flexibility and complicates failover; externalising session state is usually more scalable.

Replication and Quorums

Replication maintains multiple copies of data or service capacity. Primary–replica arrangements centralise writes and distribute reads. Multi-primary designs permit writes in multiple locations but require conflict handling. Synchronous replication limits acknowledged data loss but adds latency and can reduce availability during communication failures. Asynchronous replication improves performance and geographic reach but introduces replication lag and a possible recovery point gap.

In quorum systems, a write is acknowledged by `W` replicas and a read consults `R` replicas out of `N`. When `R + W > N`, read and write sets overlap, supporting stronger visibility assumptions. Quorums do not automatically provide complete transactional or linearizable semantics; node failure, stale reads, clock assumptions, and conflict policies still matter.

Partitioning and Sharding

Partitioning divides data or work across nodes. A good partition key distributes load evenly, supports common queries, and avoids excessive cross-partition transactions. Range partitioning supports ordered scans but can create hotspots when new records concentrate at one end. Hash partitioning distributes load more evenly but makes range queries expensive. Consistent hashing reduces reassignment when nodes join or leave.

Partitioning introduces operational complexity. Queries may require scatter-gather execution, where every partition is contacted and results are merged. Cross-partition uniqueness, joins, and transactions become more difficult. A partition key should therefore be selected from access patterns rather than from a purely conceptual data model.

Caching

Caching stores reusable results closer to consumers. Browser caches, content delivery networks, reverse proxies, application caches, and database buffer pools all reduce latency or backend load. The central difficulty is invalidation: a cached value is useful only when its freshness is acceptable.

Cache-aside design has the application read the cache first and populate it after a miss. Write-through updates the cache as part of a write path, while write-behind delays persistence and therefore requires careful durability controls. Time-to-live limits staleness but does not prevent stampedes when many requests refresh an expired item simultaneously. Request coalescing, jittered expiry, negative caching, and bounded cache sizes reduce these risks. Sensitive or user-specific data must not be accidentally shared through improperly scoped caches.

Resilience Engineering

Resilience begins with explicit failure budgets. Every remote call should have a timeout, because an indefinitely waiting request consumes threads, connections, memory, and user patience. Timeouts should be based on an end-to-end latency budget rather than copied indiscriminately across services.

Retries are appropriate for transient failures, but they amplify load during an incident. Exponential backoff with jitter spreads retry attempts over time. Retries should be limited and used mainly for idempotent operations or operations protected by an idempotency key. A payment request, for example, should carry a stable command identifier so that repeating the request does not create two charges.

A circuit breaker observes failures and temporarily stops calls to an unhealthy dependency. Its states are commonly closed, open, and half-open. Bulkheads isolate resources so that one failing dependency cannot consume all worker threads or connection-pool capacity. Rate limiting protects scarce resources and provides predictable behaviour under overload. Queue limits and backpressure prevent producers from overwhelming consumers.

Graceful degradation preserves the most valuable user functions when optional dependencies fail. A retail system may permit browsing from cached data while disabling recommendations. A collaboration system may allow reading while delaying secondary analytics. Degradation must be designed in advance; returning arbitrary or misleading data is not resilience.

Autoscaling, Capacity, and Performance Engineering

Autoscaling changes resource capacity in response to demand. CPU utilisation is easy to measure but may be a poor signal for I/O-bound services. Queue depth, request rate, concurrency, memory pressure, and latency are often more meaningful. Reactive scaling responds after a signal crosses a threshold; predictive scaling uses historical patterns or forecasts. Scaling policies need cooldown periods and stabilisation windows to avoid oscillation.

Capacity planning begins with workload characterisation. Engineers should identify peak and sustained request rates, object sizes, concurrency, dependency limits, and growth assumptions. Little’s Law provides a useful approximation:

`L = λW`

where `L` is average work in the system, `λ` is throughput, and `W` is average time in the system. If throughput is increased without controlling service time, concurrency and resource consumption rise.

Performance engineering is an empirical discipline. Baselines should be established before optimisation. Load tests should include realistic data, cache states, traffic mixtures, and failure conditions. Stress testing explores the breaking point; soak testing reveals leaks and degradation over time; chaos testing examines behaviour during controlled faults. Distributed tracing connects a user request across services, while metrics and structured logs support diagnosis. Observability should measure both technical signals and business outcomes such as successful checkout rate.

Disaster Recovery and Multi-Region Design

Disaster recovery addresses events that exceed ordinary component failure, including region loss, major data corruption, or prolonged provider disruption. The recovery time objective (RTO) is the maximum acceptable time to restore service. The recovery point objective (RPO) is the maximum acceptable amount of data loss measured in time.

A backup strategy is incomplete unless restoration is tested. Backups should be encrypted, access-controlled, versioned, geographically separated where appropriate, and protected against accidental deletion or ransomware. Recovery runbooks should specify ownership, decision authority, dependencies, verification steps, and communication procedures.

Multi-region active-passive deployment simplifies consistency but may require traffic promotion and data replication during failover. Active-active deployment can reduce failover time and improve locality, but it introduces conflict resolution, global routing, data sovereignty, and operational complexity. Disaster recovery is therefore a business decision, not merely a deployment preference.

Illustrative Case: A Regional E-Commerce Platform

Background

An online retailer operates a product catalogue, cart service, order service, payment integration, inventory system, and recommendation engine. During promotional events, traffic rises sharply. The original architecture uses a single application cluster and synchronous calls to every service on the checkout path.

Situation and Analysis

Customers report slow checkout and intermittent failures. Measurement shows that recommendation requests account for substantial tail latency, while payment-provider delays occupy application threads. A database replica is also used for catalogue reads, but replication lag causes recently changed prices to appear inconsistently. During an outage, aggressive client retries multiply traffic.

The analysis separates essential from optional work. Authorisation, inventory rules, order creation, and payment confirmation are critical. Recommendations and some catalogue enrichment are not. The team also identifies a need for idempotent order commands, bounded timeouts, and a clear recovery model.

Strategy and Solution

The retailer introduces a layer-7 load balancer with readiness checks and connection draining. Catalogue data is served through a cache-aside strategy with short, business-appropriate freshness limits; price changes actively invalidate affected entries. Checkout receives an idempotency key and stores order-command status so that retries are safe.

The recommendation call is removed from the synchronous checkout path and replaced with asynchronous event processing. Payment calls receive a strict timeout, bounded exponential backoff, and a circuit breaker. Separate connection pools and worker limits create a bulkhead around the payment integration. Inventory and order data use a consistency model strong enough to protect stock invariants, while analytics and recommendations use eventual consistency.

The platform adopts queue-based smoothing for promotional traffic, autoscaling based on queue depth and latency, and a tested backup and regional recovery procedure. Dashboards report p95 and p99 latency, checkout success, payment outcomes, queue age, replica lag, and cache effectiveness.

Outcome

The architecture no longer requires every dependency to be healthy for browsing or checkout preparation to remain available. Failures become more localised, retry amplification is reduced, and operators can distinguish dependency failure from application defects. The design does not eliminate trade-offs: stronger inventory coordination remains more expensive, and asynchronous processing requires users to understand intermediate order states.

Learning Points

The case demonstrates that resilience is achieved through deliberate dependency classification, not through replication alone. Latency budgets, idempotency, backpressure, observability, and recovery testing must be designed together. A system should preserve high-value operations while openly communicating delayed or unavailable secondary functions.

Common Errors, Risks, and Governance Concerns

A frequent mistake is adding retries without a timeout or without considering multiplication across service layers. Another is treating replicas as automatically consistent, or assuming that a health check proving process liveness proves business readiness. Unbounded queues, unlimited concurrency, shared database bottlenecks, oversized synchronous call graphs, and caches without invalidation policies are recurring sources of failure.

Distributed systems also create governance obligations. Logs and traces may contain personal or confidential data; collection should follow data-minimisation, retention, access-control, and applicable privacy requirements. Cross-border replication can raise data-residency concerns. Automated failover should have safeguards against split-brain operation and accidental destructive actions. Reliability targets must be communicated honestly: a graceful degradation mode should not falsely imply that a transaction has completed.

Current Developments and Best Practices

Service meshes increasingly provide standardised traffic policy, telemetry, identity, and mutual TLS, although they add operational layers and should not replace sound application design. Event-driven architectures and durable workflow engines support long-running processes, compensation, and asynchronous integration. Platform engineering provides reusable deployment, observability, and recovery capabilities to development teams. AI-assisted operations can help detect anomalies, forecast demand, and suggest likely causes, but automated remediation requires guardrails, explainability, and human oversight for high-impact changes.

Effective practice can be summarised as follows:

• define service-level objectives and error budgets before selecting infrastructure;
• model dependencies, latency budgets, data ownership, and failure modes explicitly;
• make commands idempotent and bound every remote operation with timeouts;
• use retries sparingly, with backoff, jitter, and admission controls;
• isolate critical resources with bulkheads, quotas, and separate pools;
• choose consistency per business invariant rather than by fashion;
• test restoration, failover, overload, and degraded modes regularly; and
• treat observability, security, privacy, and operational ownership as architectural requirements.

Chapter Summary

Distributed systems gain scale and organisational flexibility by dividing work across networked components, but they introduce latency, uncertainty, partial failure, and coordination costs. Sound architecture distinguishes availability, reliability, resilience, scalability, and performance. CAP-related reasoning helps teams choose how consistency and availability should behave during partitions, while practical consistency models allow different operations to receive different guarantees.

Load balancing, replication, partitioning, and caching support scale, but each introduces trade-offs involving routing, stale data, hotspots, conflict, and operational complexity. Timeouts, bounded retries, idempotency, circuit breakers, bulkheads, rate limits, backpressure, and graceful degradation limit fault propagation. Autoscaling and performance engineering require realistic measurement, tail-latency analysis, capacity models, and controlled testing. Disaster recovery extends these concerns to RTO, RPO, restoration, and regional failure. The central professional lesson is that resilience is a property deliberately engineered across architecture, code, operations, governance, and business priorities.

Key Terms

Distributed system: A set of networked components that cooperate to provide a service.

Partial failure: Failure of one component or communication path while other components continue operating.

Latency: Time elapsed between a request and its observable response.

Tail latency: High-percentile response time, such as p95 or p99 latency.

Availability: The proportion of time a service is usable and accessible.

Reliability: The probability that a system performs correctly over a specified period.

Resilience: The ability to withstand disruption and recover acceptably.

Scalability: The ability to handle increased workload through additional or altered capacity.

CAP theorem: The result that partitioned systems cannot guarantee both strong consistency and universal availability.

Eventual consistency: A model in which replicas converge when updates cease, without immediate agreement.

Replication: Maintaining multiple copies of data or service capacity.

Quorum: A sufficient subset of replicas required to acknowledge or answer an operation.

Partitioning: Dividing data or work across nodes or logical segments.

Cache invalidation: The process of removing or updating cached data when it is no longer suitable.

Idempotency: The property that repeating an operation produces the same intended effect.

Circuit breaker: A mechanism that temporarily blocks calls to an unhealthy dependency.

Bulkhead: Resource isolation that prevents one failure from exhausting shared capacity.

Backpressure: A control mechanism that slows producers when consumers or queues are overloaded.

RTO: Recovery time objective, or the maximum acceptable restoration time.

RPO: Recovery point objective, or the maximum acceptable period of data loss.

Review Questions

Short Answer Questions

1. Why is partial failure more difficult to handle than total system failure?

2. Distinguish availability from reliability.

3. What problem does an idempotency key solve?

4. Why are p99 latency measurements useful?

5. Define RTO and RPO.

Long Answer Questions

1. Explain CAP-related trade-offs and show why consistency requirements should be selected per business operation.

2. Compare primary–replica replication, multi-primary replication, and quorum-based approaches.

3. Discuss how timeouts, retries, circuit breakers, bulkheads, and backpressure work together.

4. Explain the benefits and risks of partitioning and caching in a high-volume application.

5. Describe a complete performance-engineering and disaster-recovery strategy for a multi-region service.

Analytical / Application Questions

1. A checkout request calls six services sequentially, retries every failed call three times, and shares one connection pool. Analyse the likely failure modes and propose a redesigned call path.

2. A catalogue system must remain available during a regional partition, while stock deduction must not oversell inventory. Propose consistency, replication, and degradation strategies.

3. An autoscaling policy based only on CPU fails during an I/O-bound traffic surge. Select better signals and explain how you would validate the revised policy.

22. Multiple Choice Questions with Answers

Question 1

Which situation best illustrates partial failure?

A. Every server is intentionally shut down for maintenance.

B. One payment provider is unreachable while browsing and catalog services continue operating.

C. A user enters an invalid password.

D. A database schema is changed successfully.

Correct Answer: B. One payment provider is unreachable while browsing and catalog services continue operating.

Question 2

During a network partition, a system that rejects some writes to preserve a strong ordering is primarily prioritising which property?

A. Availability

B. Consistency

C. Horizontal scalability

D. Cache freshness

Correct Answer: B. Consistency

Question 3

Which design most directly prevents a retried payment request from charging a customer twice?

A. Increasing the request timeout

B. Using round-robin load balancing

C. Providing an idempotency key and storing command status

D. Adding a read replica

Correct Answer: C. Providing an idempotency key and storing command status

Question 4

What is the principal danger of unlimited retries across several service layers?

A. They always improve consistency.

B. They can amplify load during an incident.

C. They eliminate the need for timeouts.

D. They prevent cache invalidation.

Correct Answer: B. They can amplify load during an incident.

Question 5

Which technique isolates payment-provider exhaustion from the rest of an application?

A. A shared unlimited connection pool

B. A bulkhead with a separate pool and bounded workers

C. A longer global timeout

D. Disabling all health checks

Correct Answer: B. A bulkhead with a separate pool and bounded workers

Question 6

Which statement about cache-aside caching is correct?

A. The database is never updated after a cache write.

B. The application reads the cache first and populates it after a miss.

C. Every cached value is automatically strongly consistent.

D. Cache-aside removes the need for expiration policies.

Correct Answer: B. The application reads the cache first and populates it after a miss.

Question 7

What does an RPO of fifteen minutes specify?

A. The maximum time allowed to restore service

B. The maximum acceptable period of data loss

C. The target p99 latency

D. The number of replicas required

Correct Answer: B. The maximum acceptable period of data loss

Question 8

A service has low average latency but unacceptable p99 latency during promotions. Which response is most appropriate?

A. Measure only the average more frequently.

B. Investigate queueing, contention, dependencies, and tail-latency causes.

C. Remove all timeouts.

D. Reduce the number of health checks.

Correct Answer: B. Investigate queueing, contention, dependencies, and tail-latency causes.

Question 9

Which partitioning strategy generally distributes keys evenly but makes range queries more difficult?

A. Hash partitioning

B. Range partitioning

C. Manual backup partitioning

D. Read-only replication

Correct Answer: A. Hash partitioning

Question 10

In the illustrative retailer case, why was the recommendation engine removed from synchronous checkout?

A. Recommendations must always be strongly consistent.

B. It was optional work whose latency and failure should not block a critical operation.

C. It required more database replicas than inventory.

D. It prevented all forms of autoscaling.

Correct Answer: B. It was optional work whose latency and failure should not block a critical operation.