9. Reliability, Resilience and Disaster Recovery

Introduction and Context

A scalable web system is not successful merely because it handles increasing traffic. It must also continue to provide an acceptable service when components fail, networks become unreliable, deployments introduce defects, or an entire operating location becomes unavailable. Reliability concerns the probability that a system performs its intended function over a specified period. Resilience is the ability to withstand disruption, recover, and continue delivering an acceptable level of service. Disaster recovery is the planned restoration of technology services and data after a severe failure.

These concerns are closely related but not identical. A reliable system seeks to prevent failure; a resilient system assumes that failure will occur and limits its consequences; disaster recovery addresses large-scale restoration when ordinary resilience mechanisms are insufficient. In distributed systems, failure is not an exceptional event. Servers crash, requests are duplicated, dependencies slow down, credentials expire, storage becomes unavailable, and software changes behave differently under production conditions. Consequently, reliability must be designed into architecture, operations, development processes, and organizational decision-making.

For a full-stack engineer, reliability is both a technical and a product responsibility. A payment service, for example, must protect transactional correctness, while a content platform may preserve core reading functions even if recommendations are unavailable. Engineering teams therefore need explicit service targets, controlled failure-handling mechanisms, tested recovery procedures, and a disciplined approach to trade-offs between availability, cost, performance, and consistency.

Learning Objectives

After completing this chapter, students should be able to:

• distinguish reliability, availability, resilience, fault tolerance, business continuity, and disaster recovery;
• design redundancy, timeouts, retries, circuit breakers, bulkheads, and graceful degradation;
• define meaningful SLIs, SLOs, SLAs, and error budgets;
• develop backup, restoration, RTO, and RPO strategies;
• analyse failure scenarios using dependency maps, risk assessment, and fault-injection techniques;
• evaluate operational, managerial, ethical, and governance implications of reliability decisions;
• apply these ideas to a scalable web architecture and a realistic incident scenario.

Reliability as a System Property

Reliability is not simply a property of an individual server. It emerges from the interaction of software, infrastructure, data, people, procedures, and external dependencies. A service may run on highly reliable machines and still be unreliable because of an unsafe deployment process or an unbounded call to a failing database.

Availability is commonly expressed as the proportion of time a service is operational and usable. If a service is unavailable for 43.2 minutes in a 30-day month, its availability is approximately 99.9 percent. Availability targets must be interpreted carefully: a service can respond successfully while returning incorrect data, and it can be technically available while being too slow for users. Reliability measurement should therefore include correctness, latency, durability, and user impact.

Fault tolerance refers to the capability to continue operation despite faults. High availability generally uses redundancy and rapid failover, whereas resilience includes adaptation, degradation, recovery, and learning. Durability concerns whether stored data survives failures. Safety concerns avoiding harmful or invalid outcomes, such as charging a customer twice. In financial systems, safety and correctness may be more important than uninterrupted availability.

A useful mental model is the chain of failure:

hazard → fault → error → failure → customer impact

A hazard is a condition capable of causing harm, such as a software defect or power interruption. A fault is the actual defect or abnormal condition. An error is an invalid internal state, and a failure occurs when externally visible service behaviour deviates from its specification. Controls such as validation, isolation, retries, and failover attempt to break this chain at different points.

Redundancy and Failure Domains

Redundancy provides more than one resource capable of performing a function. It may involve multiple application instances, database replicas, availability zones, regions, network routes, or suppliers. Redundancy is effective only when replicas do not share the same failure domain. Two servers in one rack may both fail because of a power or network problem. Two database replicas may both be corrupted by an erroneous migration.

Architects should identify failure domains explicitly: process, host, rack, zone, region, cloud provider, and organization. Active-active designs distribute traffic among functioning replicas, while active-passive designs keep a standby resource ready for promotion. Active-active systems can provide better continuity but introduce challenges involving data consistency, conflict resolution, and operational complexity. Replication does not automatically constitute backup; a destructive update may be copied to every replica.

Containing Failure in Distributed Systems

Failure containment is often more valuable than attempting to eliminate every failure. Several complementary patterns are central to resilient web architecture.

Timeouts and Deadlines

A timeout limits how long a caller waits for a response. Without timeouts, a slow dependency can consume threads, connections, memory, and user patience until the caller also fails. Timeouts should be set according to an end-to-end latency budget and the operation’s business value. A deadline should be propagated across service calls so that downstream services do not continue working after the original request can no longer succeed.

Timeouts must distinguish connection, request, read, and overall operation limits where appropriate. Very short limits cause avoidable failures; very long limits allow resource exhaustion. Measurements from production traffic should guide the choice rather than arbitrary round numbers.

Retries and Backoff

Retries can recover from transient faults such as brief packet loss or temporary overload. They can also multiply load during an incident. If many clients retry immediately, a failing dependency may receive a storm of requests. Exponential backoff increases the delay between attempts, and jitter randomizes those delays so clients do not retry simultaneously.

Retries are safe only when the operation is idempotent or protected by an idempotency key. Repeating a read is usually safe; repeating a payment request may not be. A practical retry policy specifies eligible errors, maximum attempts, total deadline, backoff, and whether the operation can be safely repeated. Retries should normally be limited to one carefully chosen layer rather than independently implemented by every layer.

Circuit Breakers

A circuit breaker monitors failures or excessive latency and changes state. In the closed state, requests flow normally. When a threshold is exceeded, it opens and rejects calls quickly, preventing continued pressure on the dependency. After a recovery interval, it enters a half-open state and permits a small number of test requests. Success closes the circuit; failure opens it again.

Circuit breakers protect callers and failing dependencies, but they do not repair the underlying problem. Thresholds, open duration, fallback behaviour, and observability must be designed carefully. A breaker that reacts to normal isolated errors may cause unnecessary outages, while one that reacts too slowly may provide no protection.

Bulkheads and Resource Isolation

Bulkheads isolate resources so that failure in one workload does not exhaust the entire system. Separate connection pools, queues, thread pools, CPU quotas, and rate limits can prevent a slow reporting task from affecting checkout requests. Isolation may be organized by tenant, endpoint, priority, or dependency.

Bulkheads involve a trade-off: unused capacity in one compartment may not be available to another. Nevertheless, protecting critical paths is often worth reduced overall utilization. Capacity limits should be paired with admission control, queue policies, and clear rejection responses.

Graceful Degradation

Graceful degradation provides a reduced but useful service when optional components fail. A shopping application may disable personalized recommendations while preserving search and checkout. A social platform may show cached content when live ranking is unavailable. Degradation should be intentional, visible where necessary, and safe: stale data may be acceptable for product descriptions but not for account balances.

Feature flags, cached responses, read-only modes, asynchronous processing, and static fallback pages are common techniques. The architecture should identify critical, important, and optional functions before an incident occurs. Otherwise, an emergency fallback may create misleading information or violate business rules.

Measuring Reliability: SLI, SLO, SLA and Error Budgets

Reliability improves when expectations are measurable. A service-level indicator (SLI) is a quantitative measurement of a user-relevant aspect of service performance. Examples include the proportion of valid requests completed successfully, the percentage of requests below a latency threshold, freshness of data, or the proportion of successfully processed jobs.

A service-level objective (SLO) is a target for an SLI over a stated period. For example, an order API might have an SLO that 99.95 percent of valid requests succeed in a calendar month, with a specified latency objective for the 95th or 99th percentile. Percentiles are generally more informative than averages because a small group of very slow requests can be hidden by a mean.

A service-level agreement (SLA) is a formal commitment to customers, often including consequences such as service credits. An internal SLO should normally be stricter than an external SLA, allowing the organization time to respond before contractual obligations are breached. These terms should not be used interchangeably.

The error budget is the permitted amount of unreliability implied by an SLO. If the target is 99.9 percent over a 30-day month, the budget is approximately 0.1 percent of measured service time or eligible requests. Error budgets create a governance mechanism: when the budget is healthy, teams may release improvements or accept controlled risk; when it is nearly exhausted, feature work may be slowed in favour of reliability work. This is not a mechanical rule. Critical security changes, legal obligations, and urgent defects may override ordinary budget policies.

Good indicators measure what users experience and exclude events that the service cannot reasonably control, while remaining resistant to manipulation. A dashboard containing only CPU and memory metrics is not a reliability measurement system. Technical telemetry should be connected to business outcomes, such as successful purchases, completed uploads, or timely notifications.

Disaster Recovery Engineering

Disaster recovery begins with business impact analysis. The organization identifies essential services, dependencies, acceptable data loss, maximum tolerable interruption, legal requirements, and recovery priorities. Recovery plans should cover people and decisions as well as technology.

The recovery time objective (RTO) is the maximum acceptable time between service disruption and restoration of service. The recovery point objective (RPO) is the maximum acceptable amount of data loss measured in time. An RPO of five minutes may require frequent replication or transaction-log shipping; an RPO of twenty-four hours may be satisfied by daily backups. Lower RTO and RPO targets generally increase cost and complexity.

A backup policy should define what is backed up, frequency, retention, encryption, access control, storage locations, immutability, ownership, and restoration testing. The commonly used principle of multiple copies across different media and locations is useful, but it is not a substitute for evidence that restoration works. Backups can be incomplete, corrupted, inaccessible, or logically contaminated by malware.

Recovery strategies include restoring from backups, rebuilding infrastructure from code, promoting a replica, switching to a secondary region, or operating in a manually reduced mode. Infrastructure as code, versioned configuration, documented runbooks, automated validation, and regular recovery exercises reduce dependence on individual employees. A plan that exists only in a document and has never been tested is an assumption, not a capability.

Chaos Engineering and Incident Learning

Chaos engineering is the disciplined practice of introducing controlled failures to test whether a system behaves as expected. Experiments may terminate instances, delay network calls, revoke access to a test dependency, fill storage, or simulate regional loss. A sound experiment begins with a hypothesis, defines steady-state indicators, limits blast radius, establishes abort conditions, and records observations.

Chaos testing should progress from low-risk environments and narrow experiments to carefully governed production tests. It is not random destruction and should never be used to demonstrate bravery. Synthetic traffic, canary users, maintenance windows, and explicit approval can reduce risk. The objective is to discover unknown dependencies, validate alerts and runbooks, and measure recovery—not to cause an outage.

Incident response normally includes detection, triage, mitigation, communication, recovery, and post-incident learning. Effective post-incident reviews are blameless but not responsibility-free: they examine decisions, system conditions, incentives, and process weaknesses. Action items should have owners, deadlines, and verification. Repeated incidents often indicate that teams are treating symptoms instead of removing structural causes.

An Applied Case: Regional Failure in an Online Learning Platform

Background

Consider LearnSphere, an illustrative online learning platform serving recorded lectures, assessments, discussion forums, and subscription payments. Its application runs in two cloud availability zones. Video files are stored in object storage, while user accounts and assessment results use a relational database. Recommendations and search indexing are non-critical supporting services.

Situation

During a peak examination period, a cloud-region network failure makes the primary database unavailable. Application instances remain running but cannot complete login or load assessment results. Automatic retries increase database connection pressure. Students experience repeated timeouts, and support channels become overloaded.

Analysis

The architecture has redundancy at the application layer but not an independent database recovery path. Retry logic is implemented in the web layer, API layer, and database client, multiplying requests. The platform has a backup policy but has tested restoration only in a development environment. Its published SLA promises high availability, while its internal SLO does not distinguish video playback from assessment submission. The most serious risks are loss of assessment data, unsafe repeated submissions, and unclear communication.

Approach and Strategy

The team first disables non-essential recommendations and search indexing, preserving resources for authentication and assessments. Circuit breakers stop calls to unavailable dependencies. The assessment API changes to a controlled read-only mode where appropriate and uses idempotency keys for submissions. A secondary database replica is promoted after validation of replication status. Requests are routed through a health-checked endpoint, while a static status page communicates the incident.

After service restoration, the team introduces a regional recovery design, separates connection pools for critical and optional workloads, and establishes a single retry policy with exponential backoff and jitter. Backups are encrypted, retained according to institutional requirements, and restored quarterly in an isolated environment. New SLOs distinguish successful assessment submissions, login availability, and video delivery. A disaster exercise simulates regional loss without exposing real student data.

Outcome

The immediate incident is contained with limited loss of non-critical functionality. The post-incident review identifies that availability alone was insufficient: data correctness, submission safety, and communication were equally important. The revised architecture improves recovery capability, but the organization accepts higher infrastructure cost and operational complexity.

Learning Points

The case demonstrates that redundancy must cover failure domains, retries require safety controls, and graceful degradation depends on prior classification of critical functions. It also shows why RTO and RPO must be linked to business processes rather than chosen as purely technical numbers.

Professional and Governance Considerations

Reliability decisions affect customers, employees, suppliers, and the public. Organizations must communicate outages accurately and promptly, avoid concealing material service failures, and preserve evidence for investigation. Personal and regulated data in backups requires encryption, least-privilege access, retention controls, and tested deletion procedures. Recovery environments must not become uncontrolled copies of sensitive production data.

Accessibility and equity also matter. A fallback that works only in a modern browser or assumes continuous broadband may exclude users. Critical services should consider alternative communication channels and support procedures. Third-party dependencies require contractual review of availability commitments, incident notification, data location, exit procedures, and evidence of recovery capability.

Automation and AI can assist with anomaly detection, capacity forecasting, incident correlation, and runbook execution. However, automated remediation can amplify an error if its signals are wrong. High-impact actions should have safeguards, audit trails, rate limits, and human escalation. An AI-generated diagnosis is a hypothesis requiring verification, not an authority that replaces engineering judgment.

Common Failure Modes and Best Practices

Common mistakes include relying on one region, treating replicas as backups, retrying non-idempotent operations, using averages instead of percentiles, setting timeouts without an end-to-end budget, and defining targets that cannot be measured. Other failures arise from untested runbooks, shared credentials, alert floods, excessive dependence on one specialist, and post-incident reviews that assign blame without changing the system.

A mature reliability programme generally follows these practices:

• map critical user journeys and their dependencies;
• assign owners for services, data, SLOs, and recovery procedures;
• design for failure domains rather than merely adding instances;
• use deadlines, bounded retries, backoff, jitter, circuit breakers, and bulkheads deliberately;
• separate critical and optional workloads and define safe fallback modes;
• monitor user-centred SLIs alongside technical indicators;
• review error budgets in release and portfolio decisions;
• maintain encrypted, isolated, versioned backups and test restoration;
• rehearse incident response, regional recovery, and communication;
• record lessons and verify that corrective actions reduce measured risk.

Current Developments

Modern platforms increasingly use multi-region deployment, managed databases, service meshes, policy-as-code, automated failover, and continuous verification of recovery readiness. These capabilities can reduce operational effort but may also hide complexity and increase dependence on providers. Platform teams are adopting reliability scorecards that combine SLO compliance, incident history, recovery-test results, dependency risk, and change safety.

The emerging emphasis is not maximum availability at any cost. It is proportionate resilience: matching investment to user harm, business criticality, regulatory exposure, and realistic failure scenarios. This approach recognizes that every system has limits and that transparent trade-offs are more sustainable than unexamined promises.

Chapter Summary

Reliable and resilient systems combine prevention, containment, degradation, recovery, and learning. Redundancy is effective only across independent failure domains. Timeouts protect resources; bounded retries with backoff and jitter address transient faults; circuit breakers stop cascading pressure; bulkheads isolate workloads; and graceful degradation preserves essential functions. SLI, SLO, SLA, and error budgets connect technical behaviour with organizational decisions. Disaster recovery requires business impact analysis, tested backups, explicit RTO and RPO targets, documented ownership, and rehearsed restoration. Chaos engineering and blameless incident learning reveal weaknesses before they become major failures. Professional practice also requires secure data handling, honest communication, accessibility, governance of suppliers, and caution when automating remediation.

Key Terms

Reliability: The probability that a system performs its intended function for a specified period under stated conditions.

Resilience: The ability to withstand disruption, recover, and continue acceptable service.

Availability: The proportion of time or eligible requests for which a service is usable.

Fault tolerance: The capacity to continue operating despite specified faults.

Failure domain: A set of resources exposed to a common cause of failure.

Redundancy: Provision of multiple resources capable of performing the same function.

Timeout: A limit on how long an operation may wait for completion.

Idempotency: A property in which repeating an operation produces the same intended effect as performing it once.

Exponential backoff: A retry strategy that progressively increases delay between attempts.

Circuit breaker: A control that stops calls to a failing dependency and later probes for recovery.

Bulkhead: Resource isolation that prevents one workload from exhausting shared capacity.

Graceful degradation: Continued delivery of essential service with optional functionality reduced.

SLI: A measured indicator of a user-relevant service characteristic.

SLO: A target value or range for an SLI over a defined period.

SLA: A formal service commitment, often contractual, made to a customer.

Error budget: The permitted unreliability implied by an SLO.

RTO: The maximum acceptable time to restore service after disruption.

RPO: The maximum acceptable amount of data loss measured in time.

Chaos engineering: Controlled experimentation that tests system behaviour under failure.

Disaster recovery: The planned restoration of services and data after a major disruption.

Review Questions

Short Answer Questions

1. Distinguish reliability, resilience, and fault tolerance.

2. Why can retries worsen an existing outage?

3. What is the difference between an SLO and an SLA?

4. Define RTO and RPO.

5. Why are replicas not a complete substitute for backups?

Long Answer Questions

1. Explain how timeouts, retries, circuit breakers, bulkheads, and graceful degradation work together to limit cascading failure.

2. Describe how an organization should design and test a disaster recovery programme for a critical web service.

3. Explain error budgets and analyse their value in release and product-management decisions.

4. Discuss redundancy across failure domains, comparing active-active and active-passive designs.

5. Describe a complete chaos-engineering experiment, including hypothesis, steady state, safeguards, and learning.

Analytical / Application Questions

1. An order service retries payment requests at three application layers. Analyse the risks and propose a safer design.

2. A service meets its availability target but has frequent incorrect responses and unacceptable p99 latency. Propose suitable SLIs and revised SLOs.

3. Design a recovery strategy for an assessment platform with an RTO of one hour and an RPO of fifteen minutes, including technical and governance controls.

22. Multiple Choice Questions with Answers

Question 1

Which design choice most directly prevents a slow dependency from exhausting caller resources?

A. Removing all monitoring

B. Using bounded timeouts and deadlines

C. Increasing retry attempts without limits

D. Replicating data within the same process

Correct Answer: B. Using bounded timeouts and deadlines

Question 2

Why should a payment operation generally use an idempotency key when retries are possible?

A. To increase payload size

B. To guarantee zero network latency

C. To prevent repeated requests from causing duplicate effects

D. To eliminate the need for authentication

Correct Answer: C. To prevent repeated requests from causing duplicate effects

Question 3

What is the principal purpose of a circuit breaker in the open state?

A. To reject calls quickly and reduce pressure on a failing dependency

B. To increase traffic to the dependency

C. To delete all dependency data

D. To bypass every security check

Correct Answer: A. To reject calls quickly and reduce pressure on a failing dependency

Question 4

An SLO of 99.9 percent over a measurement period implies that the organization has:

A. No permitted failures

B. An error budget of approximately 0.1 percent

C. A contractual SLA automatically

D. A guarantee of zero latency variation

Correct Answer: B. An error budget of approximately 0.1 percent

Question 5

Which statement correctly describes RPO?

A. The maximum time permitted to restore service

B. The percentage of requests that must succeed

C. The maximum acceptable amount of data loss measured in time

D. The time required to open a circuit breaker

Correct Answer: C. The maximum acceptable amount of data loss measured in time

Question 6

Which situation best demonstrates graceful degradation?

A. Allowing a recommendation failure to disable checkout

B. Serving cached catalogue content while preserving account payments

C. Increasing all request timeouts indefinitely

D. Deleting optional data during an outage

Correct Answer: B. Serving cached catalogue content while preserving account payments

Question 7

Why is a database replica not necessarily a sufficient backup?

A. Replicas cannot answer queries

B. A destructive update may be copied to every replica

C. Replicas always increase RPO

D. Backups cannot be encrypted

Correct Answer: B. A destructive update may be copied to every replica

Question 8

Which feature is essential to a responsible chaos-engineering experiment?

A. An unlimited blast radius

B. A hypothesis, steady-state measure, and abort condition

C. No communication with operators

D. Random failure without observation

Correct Answer: B. A hypothesis, steady-state measure, and abort condition

Question 9

What is the main purpose of a bulkhead?

A. To isolate resource consumption between workloads

B. To make every request synchronous

C. To remove all capacity limits

D. To replace disaster recovery planning

Correct Answer: A. To isolate resource consumption between workloads

Question 10

A platform has an RTO of one hour. Which statement follows?

A. It may lose at most one hour of data

B. It must restore service within one hour of disruption

C. It must maintain 100 percent availability

D. It must retain backups for exactly one hour

Correct Answer: B. It must restore service within one hour of disruption