Fault Tolerance and Resilience Engineering

Introduction

Distributed systems operate across independent machines, networks, storage devices, processes, and administrative domains. Consequently, failure is not an exceptional event but a normal operating condition. A server may crash, a network may drop or duplicate messages, a clock may be inaccurate, a storage device may corrupt data, or a compromised component may send deliberately inconsistent information. The central engineering question is therefore not whether failures will occur, but whether the system can continue to provide acceptable service when they do occur.

Fault tolerance is the ability of a system to continue performing its required functions despite faults. Resilience is broader: it includes anticipation, resistance, recovery, adaptation, and learning before, during, and after disruptive events. A resilient distributed system does not merely mask failures; it detects them, limits their propagation, recovers state, preserves safety, and restores service within defined business objectives.

This chapter develops models for crash, omission, timing, and Byzantine faults; examines redundancy and failure detection; explains checkpointing and recovery; and applies operational patterns such as circuit breakers, bulkheads, retries, timeouts, and graceful degradation. It also introduces quantitative measures of reliability, availability, recovery objectives, and resilience trade-offs.

Learning Objectives

After studying this chapter, a learner should be able to:

1. Distinguish faults, errors, failures, hazards, and incidents.

2. Model crash, omission, timing, and Byzantine behaviour in distributed systems.

3. Select suitable redundancy, replication, detection, checkpointing, and recovery techniques.

4. Calculate and interpret availability, reliability, MTTF, MTTR, RPO, and RTO.

5. Design circuit breakers, bulkheads, retry policies, and graceful-degradation strategies.

6. Evaluate resilience trade-offs involving cost, consistency, latency, complexity, and security.

7. Critically assess a fault-tolerant architecture using a structured case analysis.

1. Fault, Error, Failure, and Resilience

A fault is the underlying cause of an abnormal condition. Examples include a failed disk, a software defect, an incorrect configuration, or a malicious process. An error is an erroneous internal state produced by a fault. A failure occurs when externally observable service behaviour deviates from its specification. The causal chain is commonly represented as:

fault → error → failure.

A latent fault remains hidden until activated. An active fault produces an error, which may be masked, detected, propagated, or transformed into a failure. Fault tolerance seeks to prevent error propagation or to recover before users observe unacceptable failure.

Resilience engineering considers the complete system, including technology, operators, procedures, suppliers, and users. A resilient service has explicit service-level objectives (SLOs), observable failure modes, controlled degradation, tested recovery procedures, and mechanisms for learning from incidents.

A useful distinction is between safety and liveness. Safety means that “nothing bad happens,” such as no duplicated payment or no contradictory committed state. Liveness means that “something good eventually happens,” such as a valid request eventually receiving a response. Recovery mechanisms must preserve safety even when liveness is temporarily reduced.

2. Fault Models

A fault model states which abnormal behaviours a system is designed to tolerate. No system can be proven tolerant of an unspecified fault set. The model must identify assumptions about processes, communication, time, storage, and adversaries.

2.1 Crash faults

A crash fault occurs when a process stops executing and does not resume during the relevant interval. In a fail-stop model, the crash is eventually detected and the process performs no further actions. In a crash-recovery model, the process may restart, possibly with volatile state lost but durable state retained.

Crash faults are comparatively simple because a crashed process does not produce new conflicting outputs. However, determining whether a silent process has crashed is difficult in an asynchronous network: the same silence may result from a slow process, a delayed message, or a partition.

2.2 Omission faults

An omission fault occurs when an expected action is not performed. A sender omission means that a process fails to send a message; a receiver omission means that it fails to receive or process one. Network packet loss, dropped acknowledgements, queue overflow, and failed writes are common examples.

Omission faults require protocols to distinguish missing communication from delayed communication. Sequence numbers, acknowledgements, retransmission, idempotent operations, and durable queues are standard responses.

2.3 Timing faults

A timing fault violates a specified time bound. A process may respond too late, or it may respond too early when a protocol requires a minimum delay. Timing faults are central to real-time systems and to distributed services that use deadlines.

A timeout is evidence of a timing violation, not proof of a crash. Treating all timeouts as crashes can cause unnecessary failover, duplicated work, or cascading load. Timing assumptions must therefore be explicit: maximum processing time, message delay, clock uncertainty, and scheduling delay.

2.4 Byzantine faults

A Byzantine process can behave arbitrarily, including sending inconsistent or malicious messages to different peers. Causes include memory corruption, software compromise, insider attacks, and coordinated adversaries. Byzantine tolerance is substantially more expensive than crash tolerance because the system must establish which information is trustworthy.

In a classic authenticated Byzantine-fault model, practical Byzantine fault-tolerant replication commonly requires at least 3f + 1 replicas to tolerate f faulty replicas, assuming appropriate protocol and authentication conditions. This requirement is not a universal law for every Byzantine setting; it depends on synchrony, authentication, quorum rules, and the desired guarantees. Byzantine-resilient systems also use cryptographic authentication, quorum certificates, state-machine replication, and carefully bounded membership.

2.5 Independent and correlated failures

Availability calculations often assume independent component failures. This assumption is frequently false. A shared power supply, cloud region, software release, certificate authority, or administrator can cause correlated failure across nominally redundant components. Fault-domain diversity is therefore as important as replica count.

Common fault domains include process, virtual machine, host, rack, availability zone, region, provider, and administrative organization. Replicas placed in the same domain may provide little protection against domain-wide faults.

3. Failure Semantics and System Guarantees

Fault tolerance must be related to the service contract. A database may guarantee serializable transactions but temporarily reject requests during leader election. A payment system may preserve exactly-once business effects even if the transport provides at-least-once delivery. A media service may sacrifice completeness or resolution to preserve responsiveness.

Important delivery semantics include:

• At-most-once: an operation is attempted no more than once, but it may be lost.
• At-least-once: an operation is retried until acknowledged, so duplicates are possible.
• Exactly-once processing: each message is processed once by a component, usually through transactional mechanisms.
• Exactly-once effect: duplicate delivery may occur, but the externally visible business result occurs once, commonly through idempotency keys and deduplication.

These semantics should not be confused. Exactly-once effect is often the practical objective, while exactly-once transport is difficult and sometimes unnecessary.

4. Redundancy and Fault Masking

Redundancy supplies alternate resources when one resource fails. It can be hardware, software, data, capacity, geographic, or procedural redundancy.

4.1 Active and passive redundancy

In active replication, multiple replicas process requests concurrently. A replicated state machine uses an agreement protocol to ensure that correct replicas execute the same ordered operations. Active replication can reduce failover time but increases coordination and operational cost.

In passive replication, a primary processes requests and sends state or updates to backups. On primary failure, a backup becomes active. Passive replication may use fewer resources during normal operation, but failover can be slower and the backup may lag.

4.2 Availability of parallel components

If components fail independently and the service works when at least one of n components is operational, with component availability A_i, system availability is:

A_parallel = 1 − Π(1 − A_i).

For two identical independent components with availability A, this becomes 1 − (1 − A)^2. The formula applies only when the service truly operates with one component, failures are independent, and switching or coordination does not introduce another failure mode.

For a series arrangement, in which every component must work, availability is:

A_series = Π A_i.

These simple equations can be misleading when components share dependencies. A replicated service may still fail if all replicas depend on one network, identity service, storage system, or deployment pipeline.

4.3 Quorums

Quorum replication requires a sufficient subset of replicas to agree before an operation is accepted. For N replicas, read and write quorum sizes R and W are often selected so that R + W > N, ensuring overlap between a read and a write quorum. If strong intersection among writes is required, W must also exceed N/2. These conditions are abstract quorum conditions; actual consistency depends on conflict resolution, replica freshness, failure handling, and protocol details.

Redundancy introduces trade-offs. More replicas can increase availability and read capacity, but also increase cost, coordination traffic, stale-read risk, and the number of components that must be secured and monitored.

5. Failure Detection and Diagnosis

A failure detector provides suspicions about process or service availability. In an asynchronous distributed system, perfect failure detection is impossible because an arbitrarily slow correct process is indistinguishable from a failed process. Practical detectors therefore trade false suspicions against detection delay.

Heartbeat protocols send periodic liveness messages. If no heartbeat arrives before a timeout, a detector suspects failure. Let H be the heartbeat interval, D the maximum expected communication and processing delay under normal conditions, and M a safety margin. A rough timeout policy is T_timeout > H + D + M. This is an engineering configuration rather than a universal formula; it must be calibrated using observed latency distributions and failure objectives.

Adaptive detectors estimate normal latency and adjust suspicion thresholds. They should account for garbage collection pauses, CPU contention, network congestion, clock skew, and maintenance events. A detector should expose uncertainty rather than presenting suspicion as fact.

Diagnosis is harder than detection. A failed dependency may cause many healthy services to time out simultaneously. Causal graphs, distributed tracing, correlation identifiers, saturation metrics, and dependency-aware health checks help distinguish root cause from symptoms. Health checks should test meaningful readiness, not merely process existence. A liveness check may verify that a process is running; a readiness check verifies that it can safely receive traffic.

6. Checkpointing, Logging, and Recovery

Checkpointing periodically records sufficient state to restart computation from a known point. Without checkpoints, recovery may require replaying the entire execution history. A checkpoint can contain process memory, durable application state, metadata, and communication context.

Let C be checkpoint cost, I the average interval between checkpoints, and L the average recovery or restart cost excluding replay. Under a simplified independent-failure model with mean time between failures M, the expected lost work from a failure is approximately I/2 if failures are equally likely within an interval. The approximate overhead per unit time can therefore be expressed as:

Overhead ≈ C/I + (I/(2M)) + L/M.

The first term represents checkpoint work, the second expected recomputation, and the third recovery overhead. This expression is a simplified planning model: it assumes stable checkpoint cost, uniformly distributed failures, and negligible interaction between checkpointing and failure probability. It should not be treated as an exact prediction for highly variable workloads.

Write-ahead logging records intended state changes before applying them. After a crash, committed records can be redone and incomplete operations can be undone, depending on the logging protocol. Log sequence numbers, durable ordering, checksums, and atomic commit markers support correct recovery.

Recovery strategies include:

• Restart recovery: restart the failed component and reconstruct state.
• Rollback recovery: restore a prior checkpoint and replay subsequent operations.
• Forward recovery: correct the state without returning to an earlier checkpoint.
• Message logging: record messages so execution can be reconstructed.
• Replicated recovery: promote a synchronized replica.

Recovery must address the “recovery point consistency” problem. Restoring one component to an earlier state while another retains later effects can create impossible global state. Distributed snapshots, coordinated checkpoints, transactional logs, and replayable event streams reduce this risk.

7. Reliability, Availability, and Recovery Objectives

Reliability is the probability that a system performs without failure for a specified time. Availability is the proportion of time a service is operational and accessible according to its contract. A system can be highly available but unreliable in the sense that it fails frequently and recovers quickly; conversely, it can fail rarely but remain unavailable for a long repair.

For a simple constant-failure-rate model, reliability is:

R(t) = e^(−λt),

where λ is the constant failure rate per unit time and t is the mission duration. This exponential model assumes a memoryless process and is often inappropriate during early-life defects or component wear-out.

For a repairable system with mean time to failure MTTF and mean time to repair MTTR, steady-state availability is approximately:

A = MTTF / (MTTF + MTTR).

This assumes a repeating operational cycle, stable averages, and complete restoration after repair. If MTTF = 100 hours and MTTR = 1 hour, availability is 100/101, approximately 0.9901, or 99.01%. Reducing MTTR to 0.1 hour gives 100/100.1, approximately 99.90%. The example demonstrates why rapid detection, automation, and recovery can improve availability as effectively as reducing failure frequency.

Operational availability should include detection, diagnosis, repair, recovery, validation, and sometimes planned maintenance. Marketing availability that excludes these intervals is not a useful engineering measure.

Recovery Point Objective (RPO) is the maximum tolerable amount of data loss measured backward in time. An RPO of 15 minutes means that, after a major failure, the organization accepts losing up to 15 minutes of committed data, subject to the system’s precise definition of committed. Recovery Time Objective (RTO) is the maximum tolerable time to restore service after disruption. RPO drives replication and backup frequency; RTO drives standby capacity, automation, and restoration procedures.

Other useful measures include mean time to detect (MTTD), mean time to acknowledge, mean time to recover (MTTR), error-budget consumption, recovery success rate, and blast radius. Metrics should be defined with clear start and end events; otherwise teams may optimize measurement rather than resilience.

8. Resilience Patterns for Distributed Services

8.1 Timeouts and deadlines

Every remote call should have a bounded deadline derived from the user or upstream request deadline. An unbounded wait consumes threads, connections, memory, and queue capacity. Deadlines should propagate across service boundaries, and work that cannot complete before the deadline should be cancelled or deprioritized.

8.2 Retries and backoff

Retries help with transient faults but can amplify overload. A sound retry policy uses a small bounded number of attempts, exponential backoff, random jitter, and classification of retryable errors. Mutating operations require idempotency keys or transactional deduplication. Retrying a request after an ambiguous timeout can otherwise create duplicate effects.

8.3 Circuit breakers

A circuit breaker monitors failures or slow responses and moves through closed, open, and half-open states. In the closed state, calls proceed. In the open state, calls fail fast or use a fallback. After a cool-down period, the half-open state permits limited probes. Success closes the circuit; failure reopens it.

Circuit breakers prevent a failing dependency from consuming all caller resources. They do not repair the dependency and should not be configured solely from arbitrary error counts. Thresholds should consider error rates, latency, traffic volume, and whether failures are caused by the caller.

8.4 Bulkheads

Bulkheads isolate resources so that one workload cannot exhaust the entire service. Examples include separate thread pools, connection pools, queues, CPU quotas, tenants, and deployment cells. Isolation limits blast radius but may reduce aggregate utilization. Capacity must be allocated carefully so that a protected pool is not too small to serve essential traffic.

8.5 Load shedding and graceful degradation

When demand exceeds safe capacity, load shedding rejects or delays low-priority work. Graceful degradation preserves essential functions while reducing optional functionality. Examples include serving cached data, lowering image quality, disabling recommendations, accepting asynchronous processing, or returning a partial result.

Degradation requires a priority policy. Critical transactions, accessibility functions, safety alerts, and audit trails should not be silently sacrificed to preserve cosmetic features. Users should receive truthful status information rather than misleadingly complete responses.

8.6 Rate limiting and admission control

Rate limits protect services from overload and abusive clients. Token-bucket and leaky-bucket mechanisms regulate request rates, but the correct limit depends on workload cost, tenant fairness, and available capacity. Admission control should occur before expensive processing whenever possible.

9. A Method for Resilience Engineering

A systematic design process includes:

1. Define critical functions and unacceptable outcomes.

2. Enumerate assets, dependencies, fault domains, and trust boundaries.

3. Construct a failure-mode and effects analysis (FMEA) or hazard analysis.

4. Specify safety, availability, latency, RPO, and RTO objectives.

5. Select fault models and assumptions explicitly.

6. Design detection, containment, recovery, and degraded modes.

7. Instrument the system with meaningful telemetry.

8. Validate through fault injection, load testing, disaster-recovery exercises, and game days.

9. Review incidents without blame and update architecture and procedures.

Chaos engineering is useful when experiments are hypothesis-driven, controlled, observable, and reversible. Randomly destroying production resources without safeguards is not resilience engineering. Experiments should begin with a steady-state hypothesis, define abort conditions, limit blast radius, and verify both technical and organizational recovery.

10. Illustrative Case: Regional Failure in an Online Payments Platform

Background

Consider a fictional payment platform, PayFlow, serving merchants through APIs deployed in three geographic regions. It must preserve payment correctness, tolerate individual host failures, and restore service after a regional outage. The platform has a business RPO of near zero for committed payments and an RTO of 30 minutes for regional disaster recovery.

Problem

An identity service deployed in one region begins returning slow responses. Payment services retry aggressively. Connection pools fill, request queues grow, and healthy instances are marked unhealthy by overly short timeouts. The resulting cascade causes partial payment unavailability. Some clients retry payment submissions, creating a risk of duplicate business effects.

Analysis

The initiating event is a timing fault in a dependency. It becomes an omission-like symptom when requests time out. The principal propagation mechanisms are unbounded retries, shared connection pools, and inadequate idempotency. A simple replica count would not solve the problem because all payment workers share the same dependency and traffic policy.

Approach

PayFlow introduces the following controls:

• Client-supplied idempotency keys stored with payment outcomes.
• Deadlines propagated from API gateway to downstream calls.
• Bounded retries only for explicitly retryable failures, with exponential backoff and jitter.
• A circuit breaker for the identity dependency, with a safe cached authorization mode for selected operations.
• Separate bulkheads for merchant traffic, settlement, reporting, and administrative work.
• Priority-based load shedding that preserves payment status queries and rejects noncritical analytics first.
• Quorum-replicated payment records across independent fault domains.
• Continuous log shipping and tested regional recovery procedures.
• Alerts based on error-budget burn, queue saturation, latency, and dependency health rather than process liveness alone.

The recovery runbook promotes traffic to another region, verifies ledger consistency, replays only confirmed durable events, and reconciles ambiguous requests using idempotency records.

Outcome

The timing fault remains visible, but it no longer consumes all service capacity. Some optional features become unavailable, while core payment submission and status functions remain operational. Duplicate client retries produce the original payment result rather than new charges. Regional recovery is validated within the 30-minute RTO, and the near-zero RPO is achieved for committed records through synchronous or appropriately acknowledged cross-region replication.

Learning points

The case demonstrates that resilience is a system property, not a single mechanism. Timeouts without bounded retries can worsen failure. Replication without fault-domain diversity can provide false confidence. Exactly-once business effect depends on application-level idempotency, not merely message transport. Graceful degradation requires explicit priorities and honest communication.

11. Limitations, Governance, and Contemporary Developments

Fault-tolerant designs can introduce new failure modes. Failover may cause split-brain operation, stale data, or duplicate execution. Automated recovery can repeatedly restart a defective service and increase load. Redundancy increases energy use, software supply-chain exposure, and operational complexity. Strong consistency may reduce availability during partitions, while eventual consistency may complicate user expectations and reconciliation.

Governance must define who may authorize failover, what evidence is retained, how recovery actions are audited, and how customer and regulatory obligations are met. Financial, healthcare, and critical-infrastructure systems may require records of incidents, recovery tests, access controls, segregation of duties, and geographical restrictions on data. Privacy-preserving backups and encryption-key recovery are part of resilience, not separate concerns.

AI can assist anomaly detection, incident correlation, capacity forecasting, and runbook recommendation. However, AI-generated remediation can be unsafe when its confidence, training data, or causal reasoning is unclear. High-impact automated actions should use approval gates, policy constraints, audit logs, rollback mechanisms, and human oversight. Models themselves require resilience against data drift, adversarial input, unavailable inference services, and silent degradation.

Contemporary architectures increasingly use cell-based isolation, multi-region active-active deployment, confidential computing, software supply-chain verification, and continuous verification of recovery procedures. These approaches do not remove the need for explicit fault models and disciplined testing.

Best Practices

• State the fault model and assumptions before selecting a protocol.
• Separate safety requirements from availability preferences.
• Design for correlated failures and independent fault domains.
• Use deadlines, bounded retries, jitter, and idempotency for remote calls.
• Isolate critical workloads with bulkheads and admission control.
• Make degraded modes explicit, prioritized, observable, and truthful.
• Measure MTTD, MTTR, RPO, RTO, error-budget burn, and recovery success.
• Test backups by restoring them and test failover under realistic load.
• Treat monitoring, runbooks, access controls, and trained operators as system components.
• Review resilience after incidents and after major architectural changes.

Chapter Summary

Fault tolerance enables continued service despite faults, while resilience includes anticipation, containment, recovery, adaptation, and learning. Crash, omission, timing, and Byzantine faults require different assumptions and controls. Redundancy, quorums, heartbeats, checkpointing, logging, and replication provide complementary protection, but their effectiveness depends on failure independence, consistency requirements, and correct recovery protocols.

Reliability describes failure-free operation over time; availability includes restoration and repair. MTTF, MTTR, RPO, RTO, and error budgets translate technical behaviour into operational and business objectives. Circuit breakers, bulkheads, timeouts, bounded retries, rate limits, load shedding, and graceful degradation limit cascading failure. A complete resilience strategy combines architecture, observability, security, governance, tested procedures, and human judgment. No mechanism eliminates risk; good engineering makes assumptions explicit, limits blast radius, preserves critical safety properties, and recovers predictably.

Key Terms

Fault: An underlying cause of abnormal system behaviour.

Error: An incorrect internal state caused by a fault.

Failure: Observable deviation from the specified service.

Fault tolerance: Ability to continue required service despite faults.

Resilience: Ability to anticipate, withstand, recover from, and adapt to disruption.

Crash fault: A process stops executing.

Omission fault: An expected send, receive, or processing action is missing.

Timing fault: A response violates a specified time constraint.

Byzantine fault: Arbitrary or malicious behaviour by a component.

Redundancy: Additional resources used to tolerate failures.

Quorum: A sufficient subset of replicas required for an operation or decision.

Failure detector: A mechanism that reports or suspects unavailable components.

Checkpoint: A saved state from which computation can restart.

MTTF: Mean time to failure.

MTTR: Mean time to repair or recover.

RPO: Maximum acceptable amount of data loss measured in time.

RTO: Maximum acceptable time to restore service.

Circuit breaker: A fail-fast control that temporarily stops calls to an unhealthy dependency.

Bulkhead: Resource isolation that limits failure propagation.

Graceful degradation: Preserving essential service while reducing optional functionality.

Review Questions

Short Answer Questions

1. Distinguish a fault, an error, and a failure.

2. Why is a timeout not proof that a process has crashed?

3. Define RPO and RTO.

4. What is the purpose of a circuit breaker?

5. Why are idempotency keys important when retries are used?

Long Answer Questions

1. Compare crash, omission, timing, and Byzantine fault models, including suitable mitigation techniques.

2. Explain active and passive replication and evaluate their principal trade-offs.

3. Describe checkpointing, write-ahead logging, and replay-based recovery.

4. Discuss how bulkheads, load shedding, and graceful degradation prevent cascading failure.

5. Explain why independent replica placement and correlated-failure analysis are essential for availability.

Analytical / Application Questions

1. A service has an MTTF of 200 hours and an MTTR of 2 hours. Calculate its approximate steady-state availability and explain how reducing MTTR affects the result.

2. Design a retry, timeout, circuit-breaker, and idempotency policy for a payment API whose downstream authorization service experiences intermittent delays.

3. A three-region system loses one region, while the remaining regions have sufficient capacity but share a vulnerable identity provider. Analyze the resilience limitations and propose architectural and operational improvements.

Multiple Choice Questions with Answers

Question 1

Which fault model permits a component to send inconsistent or malicious messages to different peers?

A. Crash fault

B. Omission fault

C. Timing fault

D. Byzantine fault

Correct Answer: D. Byzantine fault

Question 2

For two independent identical components arranged in parallel, each with availability A, which expression gives system availability?

A. A²

B. 1 − (1 − A)²

C. 2A

D. 1 − A²

Correct Answer: B. 1 − (1 − A)²

Question 3

What is the principal purpose of an idempotency key in a payment request?

A. To increase network bandwidth

B. To prevent duplicate business effects from retries

C. To eliminate the need for authentication

D. To guarantee zero latency

Correct Answer: B. To prevent duplicate business effects from retries

Question 4

In the circuit-breaker pattern, what normally happens in the open state?

A. All calls proceed normally

B. Only database writes proceed

C. Calls fail fast or use a fallback

D. The dependency is automatically reinstalled

Correct Answer: C. Calls fail fast or use a fallback

Question 5

A system has MTTF = 100 hours and MTTR = 1 hour. Under the stated steady-state approximation, what is its availability?

A. 50.0%

B. 90.0%

C. 99.01%

D. 99.99%

Correct Answer: C. 99.01%

Question 6

What does an RPO of 15 minutes specify?

A. The maximum time allowed to detect a fault

B. The maximum acceptable amount of data loss measured backward in time

C. The maximum request latency

D. The minimum number of replicas

Correct Answer: B. The maximum acceptable amount of data loss measured backward in time

Question 7

Which practice most directly limits one workload from exhausting another workload’s resources?

A. Bulkheading

B. Unbounded retrying

C. Disabling health checks

D. Removing admission control

Correct Answer: A. Bulkheading

Question 8

Why are heartbeats and timeouts unable to provide perfect failure detection in a fully asynchronous system?

A. Messages cannot be encrypted

B. A slow correct process can appear identical to a failed process

C. Replicas always have identical clocks

D. Heartbeats require Byzantine faults

Correct Answer: B. A slow correct process can appear identical to a failed process

Question 9

Which strategy best represents graceful degradation?

A. Serving cached essential data while disabling recommendations

B. Retrying every failed request indefinitely

C. Removing all replicas during overload

D. Treating every timeout as a permanent crash

Correct Answer: A. Serving cached essential data while disabling recommendations

Question 10

Under the commonly used authenticated Byzantine replication model, how many replicas are commonly required to tolerate f Byzantine replicas?

A. f

B. 2f

C. 3f + 1

D. f²

Correct Answer: C. 3f + 1