Reliability, Availability and Disaster Recovery

Introduction and Context

Cloud systems are expected to remain dependable despite hardware failure, software defects, network disruption, human error, cyberattack, and regional disasters. Reliability, availability, and disaster recovery are related but distinct disciplines. Reliability concerns whether a system performs correctly over time. Availability concerns whether the system is usable when requested. Disaster recovery concerns restoring service and data after a disruptive event.

Modern cloud architecture achieves resilience through elimination of single points of failure, redundancy, automated detection, controlled failover, tested recovery procedures, and continuous measurement. These mechanisms must be aligned with business requirements. A payment platform may require near-continuous availability and almost no data loss, whereas an internal reporting system may accept several hours of downtime.

This chapter develops the engineering concepts, calculations, design patterns, and operational practices required to build fault-tolerant cloud systems.

Learning Objectives

By the end of this chapter, a learner should be able to:

• distinguish reliability, availability, resilience, business continuity, and disaster recovery;
• design redundancy across instances, zones, regions, networks, and data stores;
• use health checks, autoscaling, load balancing, and automated failover appropriately;
• calculate SLIs, SLOs, SLAs, error budgets, RTO, RPO, MTBF, and MTTR;
• compare backup, replication, active-passive, and active-active recovery strategies;
• plan and conduct resilience, failover, backup-restoration, and disaster-recovery tests;
• evaluate security, governance, cost, and operational trade-offs in resilience design.

1. Core Concepts

Reliability

Reliability is the probability that a system performs its required function without failure for a specified period under stated conditions. A reliable application returns correct responses, preserves data integrity, and behaves predictably under normal and abnormal loads.

Reliability is not identical to uptime. A service that responds quickly but returns incorrect financial balances is available but unreliable from the customer’s perspective.

Availability

Availability is the proportion of time that a service is operational and usable according to an agreed definition. A basic steady-state model is:

Here, A is availability, MTBF is mean time between failures, and MTTR is mean time to repair or restore. Both time values must use the same units. This formula assumes repeated operating and repair cycles and does not fully represent planned maintenance, correlated failures, or partial service degradation.

For an observed measurement period:

where T is the total observation time and D is qualifying downtime. If a service is unavailable for 43.2 minutes during a 30-day month of 43,200 minutes, availability is (43,200 − 43.2) ÷ 43,200 = 0.999, or 99.9%.

Availability percentages can conceal significant operational differences. Approximate monthly downtime allowances are:

These values assume a 30-day month and continuous service measurement.

Resilience and fault tolerance

Resilience is the ability to continue delivering an acceptable service during disruption and to recover efficiently afterward. Fault tolerance is a stronger property: the system continues operation despite specified component failures, often without visible interruption.

A resilient architecture may degrade gracefully. For example, a retail service may continue browsing and order placement while temporarily disabling product recommendations. Graceful degradation protects critical functions instead of allowing a minor dependency failure to cause total outage.

Failure domains

A failure domain is a set of components likely to fail together. Common domains include:

• process or container;
• virtual machine or host;
• rack or availability zone;
• data centre or region;
• cloud provider;
• identity, DNS, certificate, or control-plane dependency;
• organisation, supplier, or human administration.

Redundancy is valuable only when replicas are placed in genuinely independent failure domains. Two virtual machines in the same rack are not equivalent to two zones.

2. Reliability Architecture Patterns

Redundancy and replication

Redundancy provides more than one resource capable of serving a function. It may be implemented as:

• N+1 capacity: required capacity plus one spare unit;
• active-passive: one component serves traffic while another is ready to take over;
• active-active: multiple components serve traffic simultaneously;
• synchronous replication: a write is confirmed after replicas acknowledge it;
• asynchronous replication: a replica receives changes later, improving distance and performance but creating possible data loss.

Redundancy must include dependencies. A redundant application tier connected to one database, load balancer, network route, or credentials service still contains a single point of failure.

For independent components that must all operate successfully, system availability is approximately:

where A_i is the availability of component i. This applies to a serial dependency path and assumes statistically independent failures. For two independent redundant components where either can serve, availability is:

For two components each with availability 0.99, redundant availability is 1 − (0.01 × 0.01) = 0.9999, or 99.99%. Real systems may perform worse because failures are correlated, failover may fail, and shared dependencies may remain.

Load balancing

Load balancers distribute requests among healthy instances. They improve availability by removing failed instances from service and supporting horizontal scaling. Important design decisions include:

• health-check endpoint and checking frequency;
• connection draining during replacement;
• session handling, preferably through stateless services or shared session storage;
• timeout, retry, and circuit-breaker policies;
• protection against retry storms;
• zonal and regional traffic distribution.

A health check should test the minimum conditions required to serve traffic. A shallow process check may report success while the application cannot reach its database. Conversely, a health check that depends on every downstream service may remove all instances during a dependency outage. Separate liveness, readiness, and dependency checks are often appropriate.

Autoscaling

Autoscaling adjusts capacity according to demand or health signals. Horizontal scaling adds instances; vertical scaling increases resources on existing instances. Horizontal scaling usually provides better fault isolation, but only when the application is designed to be distributed.

Autoscaling policies should include minimum, maximum, and desired capacity, scale-out and scale-in thresholds, cooldown or stabilization periods, and protection against oscillation. Capacity based only on CPU may be unsuitable for queue-driven or I/O-bound workloads. Queue depth, request latency, concurrency, and error rate may be better signals.

Autoscaling is not a substitute for capacity planning. If a database, quota, network, or regional limit cannot scale with the application tier, scaling application instances may increase failure.

Timeouts, retries, and circuit breakers

A timeout bounds how long a request waits for a response. Retries can overcome transient faults, but uncontrolled retries multiply load. A retry policy should define a maximum attempt count, exponential backoff, random jitter, and which errors are retryable. Non-idempotent operations require idempotency keys or other safeguards to prevent duplicate effects.

A circuit breaker changes behaviour after repeated failures. In the open state, calls fail quickly; after a delay, a limited number of test calls are permitted in the half-open state. If recovery succeeds, the circuit closes. Circuit breakers protect both callers and failing dependencies.

Queues and graceful degradation

Queues decouple producers and consumers and absorb temporary demand peaks. They require controls for message visibility, duplication, ordering, dead-letter handling, retention, and poison messages. At-least-once delivery means consumers must usually be idempotent.

Graceful degradation can include cached data, read-only operation, reduced feature sets, static responses, or admission control. Degradation rules should be designed before an incident rather than improvised during one.

3. Availability Measurement and Service Objectives

SLIs

A service-level indicator (SLI) is a measured quantity describing service behaviour. Typical SLIs include:

• successful request ratio;
• latency at the 95th or 99th percentile;
• freshness of data;
• availability of a critical workflow;
• queue processing delay;
• durability or integrity of stored data.

A ratio SLI can be expressed as:

The definition of “good” and “valid” must be explicit. For example, a successful transaction SLI might exclude requests rejected by documented client validation but include server-side errors.

SLOs and SLAs

A service-level objective (SLO) is an internal or customer-facing target, such as “99.95% of valid API requests succeed each calendar month.” A service-level agreement (SLA) is a contractual commitment, often with service credits or other remedies. An SLA should not automatically be treated as the engineering target; teams commonly set an SLO stricter than the SLA.

Error budgets

The error budget is the permitted unreliability during the SLO period:

For a 99.9% monthly availability SLO and a 43,200-minute month, the budget is 0.001 × 43,200 = 43.2 minutes. If measured downtime reaches 30 minutes, 13.2 minutes remain. Error budgets support balanced decisions: when reliability is within budget, teams may release features or conduct controlled change; when the budget is exhausted, risky changes should be restricted until reliability improves.

Error budgets are meaningful only when measurement windows, exclusions, maintenance rules, and incident classification are documented. Excluding too many events makes the objective misleading.

4. Disaster Recovery Engineering

Business impact analysis

A business impact analysis identifies critical services, consequences of interruption, dependencies, maximum tolerable downtime, data-loss tolerance, legal obligations, and recovery priorities. Recovery design should begin with business impact rather than with a particular cloud product.

RTO and RPO

Recovery time objective (RTO) is the maximum acceptable time between service interruption and restoration of an agreed service level. Recovery point objective (RPO) is the maximum acceptable period of data loss measured backward from the disruption.

If an application has an RTO of one hour, recovery should complete within one hour of the incident according to the defined measurement boundary. If its RPO is 15 minutes, the recovery process must normally restore data to a point no more than 15 minutes before the failure. RPO is not the same as backup frequency alone: restore delay, replication lag, and transaction consistency also matter.

Recovery strategies

Backup and replication are complementary. Replication can copy corruption or accidental deletion quickly; backups provide historical recovery points. A strong design uses immutable or write-once backups, separate credentials, encryption, retention policies, and regular restoration tests.

Multi-zone and multi-region architecture

A multi-zone system distributes components across isolated availability zones within a region. It commonly protects against host, rack, or zone failure while retaining relatively low latency. A multi-region system distributes service across geographically separate regions and can address regional outages, but introduces higher latency, replication complexity, data-residency issues, and more difficult operations.

Cross-region designs must address DNS or global traffic management, certificate and identity availability, database conflict resolution, clock and ordering assumptions, deployment consistency, and regional quota capacity. Geographic separation alone does not guarantee recovery if both regions share a compromised identity account or deployment pipeline.

Backups and restoration

A backup policy should specify what is backed up, frequency, retention, encryption, ownership, storage location, immutability, deletion protection, and restoration priority. The 3-2-1 principle recommends at least three copies, on two types of storage or media, with one copy geographically separate. It is a practical guideline rather than a universal guarantee.

Restoration must be tested with realistic data and dependencies. A backup that cannot be restored within the RTO is not an adequate recovery solution. Restoration tests should verify application consistency, permissions, secrets, network routes, data integrity, and post-recovery monitoring.

5. Quantitative Reliability Planning

MTBF and MTTR

Mean time between failures estimates the average operating time between failures in a repairable system. Mean time to repair or restore estimates the average time required to return it to service. They are statistical measures, not guarantees for a particular incident.

Suppose a service records 4,000 operating hours and 5 failures, with 20 total hours spent restoring service. Then:

The steady-state availability estimate is:

This estimate assumes the observations are representative and that failure and repair cycles are comparable. It should not be used to promise that the next outage will last four hours.

Availability allocation

An availability objective must be allocated across dependencies. If four independent serial components must each meet the same availability target and the overall target is 99.99%, a simple equal allocation is:

Thus A_component = 0.9999^(1/4) ≈ 0.999975, or approximately 99.9975% per component. This is only a starting allocation. Critical components may receive stricter targets, and common-mode failures must be analysed separately.

Risk and failure analysis

Failure Mode and Effects Analysis (FMEA) identifies failure modes, causes, effects, detection controls, and mitigations. A risk priority number is sometimes calculated as:

This ordinal scoring method can help prioritise workshops, but it is not a probability model. Scores are subjective and should not replace quantitative loss analysis or engineering judgement.

Fault-tree analysis starts with a top event, such as “service unavailable,” and decomposes it into contributing events using logical AND and OR relationships. These methods reveal hidden shared dependencies and support design review.

6. Observability, Incident Response, and Operations

Reliability engineering depends on observability. Metrics show numerical behaviour, logs provide event detail, and traces reveal request paths across services. Useful signals include traffic, latency, errors, saturation, replication lag, queue depth, backup age, restore duration, and health-check state.

Alerting should be actionable and tied to user impact or imminent risk. Alerts based only on infrastructure symptoms can produce noise, while alerts based only on complete outage may be too late. SLO-based alerting can use fast-burn and slow-burn windows to detect rapid or gradual budget consumption.

An incident response plan should define severity levels, incident commander, technical lead, communications lead, escalation paths, status updates, decision authority, and criteria for recovery. Runbooks must contain commands or procedures that are safe, version-controlled, access-controlled, and tested.

After an incident, a blameless review should identify technical and organisational causes, contributing conditions, detection gaps, recovery delays, and specific owners and deadlines. Blameless does not mean consequence-free; it means the investigation focuses on improving systems rather than hiding information or assigning simplistic personal blame.

7. Resilience Testing

Resilience is an empirical property. It should be tested rather than inferred from architecture diagrams.

A progressive test programme may include:

1. backup restoration in an isolated environment;
2. instance termination and replacement;
3. zone evacuation or controlled traffic shift;
4. database failover and replica promotion;
5. dependency latency, throttling, and error injection;
6. regional recovery using documented procedures;
7. game days involving technical and business stakeholders.

Tests require a hypothesis, scope, safeguards, success criteria, observability, rollback plan, and post-test review. Production experiments should begin with low blast radius and explicit approval. Chaos engineering is not random destruction; it is controlled experimentation designed to validate resilience assumptions.

Important test results include achieved RTO, achieved RPO, percentage of requests preserved, data integrity, alert time, human decision time, and undocumented manual steps. A recovery plan that succeeds only because an experienced engineer remembers an undocumented action is operationally fragile.

8. Security, Governance, and Ethical Considerations

Availability controls must not weaken confidentiality or integrity. Backups should be encrypted in transit and at rest, protected against ransomware, separated from production credentials, and monitored for unusual deletion. Administrative access should use least privilege, multifactor authentication, logging, and preferably separate break-glass procedures.

Regulatory requirements may impose retention, sovereignty, privacy, notification, and recoverability obligations. Multi-region replication may transfer personal data across jurisdictions. Recovery environments must therefore be included in privacy assessments and access reviews.

Reliability decisions also have ethical consequences. An outage in a hospital, emergency service, financial platform, or public infrastructure can cause physical or social harm. Teams should prioritise critical users, provide honest status communication, avoid misleading availability claims, and test accessibility and degraded modes for vulnerable users.

Artificial intelligence can assist anomaly detection, capacity forecasting, incident correlation, and runbook search. However, automated remediation must have bounded permissions, audit trails, confidence thresholds, and human escalation. An AI system that restarts healthy services or deletes data based on a false diagnosis can amplify an incident.

9. Illustrative Case: Regional Failure of an Online Retailer

Background

A fictional online retailer operates a cloud application consisting of a web tier, order service, payment integration, inventory database, and asynchronous fulfilment workers. The business requires an RTO of 60 minutes and an RPO of 15 minutes for confirmed orders.

Problem

The initial design uses multiple virtual machines in one availability zone. A zone networking failure makes the website inaccessible for 90 minutes. Although the database is backed up nightly, restoring the previous night’s backup would lose many hours of orders. Monitoring reports that virtual machines are running, so no effective traffic failover occurs.

Analysis

The principal weaknesses are a single failure domain, shallow health checks, excessive backup interval, and lack of a tested recovery procedure. The architecture also lacks a clear distinction between payment confirmation and fulfilment processing.

Approach and solution

The retailer deploys web and order-service instances across three zones behind a regional load balancer. Read replicas and a highly available primary database are used, with transaction logs replicated to a separate region every five minutes. Backups are encrypted, immutable, and retained for 35 days.

A warm standby environment is created in a second region. Infrastructure is defined as code, and configuration and secrets are replicated through controlled processes. Global traffic management directs users to the healthy region. Readiness checks verify application dependencies, while liveness checks determine whether a process should be restarted.

Orders receive idempotency keys. Confirmed orders are written transactionally before fulfilment messages are published. The fulfilment queue is replayable, and consumers record processed message identifiers. During regional recovery, the retailer temporarily disables recommendations and some administrative reports but preserves browsing, checkout, and order status.

Outcome

A controlled regional failover test restores critical service in 42 minutes and recovers data to a point seven minutes before the simulated failure. The achieved results meet the 60-minute RTO and 15-minute RPO. The test also identifies a missing payment-provider allow-list entry, which is corrected before a real incident.

Learning points

The case demonstrates that resilience is an end-to-end property. Multi-zone compute alone would not solve data recovery. Backups alone would not satisfy the RPO. Health checks, idempotency, dependency configuration, restoration testing, and degraded-mode design are all necessary.

10. Best-Practice Design Checklist

• Identify business-critical user journeys and define measurable SLIs.
• Set SLOs, SLAs, RTO, and RPO with explicit measurement boundaries.
• Remove single points of failure across application, data, network, identity, and operations.
• Distribute replicas across independent failure domains.
• Use meaningful readiness and liveness checks.
• Design retries with bounded exponential backoff and jitter.
• Make distributed operations idempotent where duplicate execution is possible.
• Combine replication with immutable, independently protected backups.
• Monitor error-budget consumption, replication lag, backup age, and restore performance.
• Test failover and restoration regularly, including human procedures.
• Keep recovery infrastructure and documentation under version control.
• Review cost, data sovereignty, security, and environmental impact.
• After incidents and tests, convert findings into owned, time-bound improvements.

Chapter Summary

Reliability means correct operation over time; availability measures usable service; resilience describes continued operation and recovery during disruption; and disaster recovery focuses on restoring systems and data after major failure. Fault-tolerant cloud architecture uses redundancy, independent failure domains, health checks, load balancing, autoscaling, bounded retries, queues, graceful degradation, and carefully designed data replication.

SLIs measure service behaviour, SLOs express engineering targets, and SLAs express contractual commitments. Error budgets quantify permitted unreliability and help balance delivery with operational risk. MTBF and MTTR support availability estimation, while RTO and RPO define recovery time and acceptable data loss. Backup, warm standby, active-passive, and active-active strategies provide different cost, complexity, and recovery characteristics.

No architecture is resilient merely because it contains replicas. Correlated failures, shared credentials, common deployment pipelines, data corruption, and untested procedures can defeat redundancy. Observability, incident response, security controls, restoration tests, and controlled resilience experiments are therefore essential. Effective reliability engineering aligns technical design with business impact and continuously validates that stated recovery objectives can actually be achieved.

Key Terms

Reliability: Probability that a system performs its required function without failure for a specified time.

Availability: Proportion of time a service is operational and usable.

Resilience: Ability to withstand disruption, degrade acceptably, and recover.

Fault tolerance: Continued operation despite specified component failures.

Failure domain: Group of components likely to fail together.

Redundancy: Provision of multiple resources capable of performing a function.

Active-active: Multiple service instances or sites serve traffic simultaneously.

Active-passive: One primary service site operates while another is prepared for takeover.

Health check: Automated test used to determine service or component condition.

Autoscaling: Automatic adjustment of resource capacity in response to demand or health.

SLI: Quantitative indicator of service performance or reliability.

SLO: Target value or range for an SLI.

SLA: Contractual service commitment and associated remedies.

Error budget: Permitted unreliability implied by an SLO.

MTBF: Mean operating time between failures in a repairable system.

MTTR: Mean time to repair or restore service.

RTO: Maximum acceptable time to restore an agreed service level.

RPO: Maximum acceptable period of data loss measured backward from failure.

Graceful degradation: Continued delivery of essential functions with reduced features.

Chaos engineering: Controlled experimentation used to test resilience assumptions.

Review Questions

Short Answer Questions

1. Distinguish reliability from availability.
2. What is the difference between an SLI and an SLO?
3. Define RTO and RPO.
4. Why are immutable backups valuable during ransomware attacks?
5. What purpose does a circuit breaker serve?

Long Answer Questions

1. Explain how multi-zone and multi-region architectures address different failure domains.
2. Compare backup-and-restore, warm standby, active-passive, and active-active recovery strategies.
3. Discuss how health checks, load balancing, and autoscaling work together to improve availability.
4. Explain error budgets and evaluate their value in release-management decisions.
5. Design a resilience-testing programme for a cloud-based financial application.

Analytical / Application Questions

1. A service experiences 12 hours of downtime during 8,760 operating hours and has 12 failures. Calculate MTBF, MTTR, and estimated steady-state availability, stating assumptions.
2. A service has a 99.95% monthly availability SLO in a 30-day month. Calculate its error budget in minutes and explain how a team should respond after consuming 80% of it.
3. Design a recovery architecture for an application with an RTO of 30 minutes and an RPO of five minutes. Justify the replication, backup, failover, security, and testing choices.

Multiple Choice Questions with Answers

Question 1

Which metric represents the maximum acceptable period of data loss measured backward from a disruption?

A. MTTR

B. RTO

C. RPO

D. MTBF

Correct Answer: C. RPO

Question 2

A service has 43.2 minutes of downtime in a 30-day month of 43,200 minutes. What is its measured availability?

A. 99%

B. 99.9%

C. 99.99%

D. 99.999%

Correct Answer: B. 99.9%

Question 3

Which design most directly protects against failure of an entire availability zone?

A. Two processes on one virtual machine

B. Multiple instances distributed across zones

C. A larger instance in one zone

D. A longer request timeout

Correct Answer: B. Multiple instances distributed across zones

Question 4

What is the principal danger of unbounded retries?

A. They eliminate all transient errors

B. They can amplify load during an outage

C. They guarantee exactly-once processing

D. They prevent circuit breakers from opening

Correct Answer: B. They can amplify load during an outage

Question 5

For a 99.9% SLO, what is the monthly error budget in a 30-day month?

A. 4.32 minutes

B. 13.2 minutes

C. 43.2 minutes

D. 432 minutes

Correct Answer: C. 43.2 minutes

Question 6

Which statement best describes an SLA?

A. A measured service indicator

B. A contractual service commitment

C. A backup retention schedule

D. A database replication method

Correct Answer: B. A contractual service commitment

Question 7

Why should backups and replication both be used?

A. Replication always preserves historical versions

B. Backups always provide zero data loss

C. Replication supports continuity while backups support historical recovery

D. They are identical controls with different names

Correct Answer: C. Replication supports continuity while backups support historical recovery

Question 8

A system has two independent redundant components, each with availability 0.99. What is their combined availability when either can serve?

A. 0.9801

B. 0.99

C. 0.9999

D. 1.98

Correct Answer: C. 0.9999

Question 9

Which practice best reduces duplicate effects when a message is delivered more than once?

A. Making consumers idempotent

B. Removing all queues

C. Increasing DNS TTL

D. Disabling monitoring

Correct Answer: A. Making consumers idempotent

Question 10

What is the primary purpose of a resilience test or game day?

A. To prove that failure is impossible

B. To validate recovery assumptions and procedures

C. To replace all backups

D. To increase application feature count

Correct Answer: B. To validate recovery assumptions and procedures