Reliability, Performance and Observability

Introduction

Modern software systems are judged not only by whether they produce correct results, but also by whether they remain available, responsive, recoverable, and understandable under changing conditions. A banking service that returns correct balances but frequently times out is not reliable. A web application that performs well for 100 users but collapses at 10,000 users is not scalable. A distributed system whose failures cannot be diagnosed is operationally immature, even if its architecture appears sophisticated.

Reliability, performance, and observability are therefore closely connected engineering concerns. Reliability asks whether a system performs its intended function over time and under stated conditions. Performance concerns the speed and resource efficiency of that function. Observability concerns the ability to infer internal system state from externally collected evidence such as logs, metrics, and traces. Together, these disciplines support resilience: the capacity to continue providing an acceptable service despite faults, overload, dependency failures, and changing environments.

This chapter presents methods for measuring and improving these properties. It covers profiling, load testing, capacity planning, fault tolerance, chaos testing, service-level objectives, incident management, and the design of useful telemetry. The central principle is empirical engineering: claims about performance and reliability should be supported by representative measurements, explicit assumptions, and controlled experiments.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish reliability, availability, performance, scalability, resilience, and observability.
2. Select suitable service-level indicators and formulate service-level objectives.
3. Interpret latency distributions, throughput, errors, saturation, and resource profiles.
4. Design profiling, load-testing, capacity-planning, and resilience experiments.
5. Explain fault-tolerance patterns and their trade-offs.
6. Use logs, metrics, and distributed traces to diagnose incidents.
7. Apply basic quantitative models such as Little’s Law and availability calculations.
8. Participate effectively in incident response, post-incident review, and continuous improvement.

1. Reliability as a System Property

Reliability is the probability that a system performs its required function without failure for a specified time, environment, and workload. This definition is more precise than saying that a system is “stable.” It identifies four necessary elements: the intended function, the operating conditions, the time interval, and what counts as failure.

Several related terms should be separated:

• Availability is the proportion of time a service is usable when it is needed.
• Durability is the probability that stored data remains correct and recoverable.
• Maintainability is the ease and speed with which a system can be repaired or changed.
• Resilience is the ability to absorb disruption, degrade acceptably, recover, and learn.
• Scalability is the ability to handle increased demand by adding resources or changing the design.
• Performance describes response speed, capacity, and resource consumption.

A system can be highly available but unreliable in another sense. For example, it may always respond with an error message rather than correctly process transactions. Reliability must therefore use meaningful success criteria, not merely process uptime.

Availability and failure metrics

For a service observed over a period, a basic availability estimate is:

where A is availability, U is usable service time, and D is downtime. Both U and D must use the same time unit. This is a descriptive estimate, not a guarantee; it assumes that the observation period and classification of downtime are well defined.

Mean time between failures and mean time to repair are useful operational measures. Under a simple alternating failure–repair model:

MTBF is mean time between failures and MTTR is mean time to repair. The approximation is most useful when failures and repairs are reasonably stable and the system alternates between an operating state and a repair state. It does not capture correlated failures, planned maintenance, partial availability, or complex repair dependencies.

For independent components connected in series, where every component must work, approximate system availability is:

where A_i is the availability of component i. For redundant components operating in parallel, where at least one must work:

These formulas assume statistical independence and identical definitions of availability. Independence is often unrealistic: a shared power supply, deployment, database, or network can cause common-mode failure.

Reliability targets and error budgets

A service-level objective (SLO) is a target for a measurable service property over a specified period. A service-level indicator (SLI) is the measurement used to evaluate that property. A service-level agreement (SLA) is a formal commitment, often contractual, that may include consequences when targets are missed.

For example:

• SLI: proportion of valid API requests completed successfully within 300 ms.
• SLO: at least 99.9% over a calendar month.
• SLA: a customer contract promising 99.5% availability with service credits.

An error budget is the permitted amount of unsuccessful service. If the SLO is 99.9%, the error-budget fraction is:

For a 30-day month, assuming 30 × 24 × 60 = 43,200 minutes:

This calculation applies only when the SLO is an availability-style time objective. A request-based SLO should use the number of eligible requests rather than minutes. Error budgets create a decision framework: when the budget is being consumed rapidly, teams may pause risky releases and prioritize reliability work; when service is healthy, controlled innovation becomes more acceptable.

2. Performance Models and Measurement

Performance is multidimensional. A system may have high throughput but poor tail latency, low average latency but frequent timeouts, or efficient CPU use but excessive memory consumption. At minimum, performance analysis should consider:

• Latency: elapsed time for an operation, commonly reported at p50, p90, p95, p99, and p99.9.
• Throughput: completed operations per unit time, such as requests per second.
• Concurrency: operations in progress at a given time.
• Saturation: the degree to which a resource approaches a limiting capacity.
• Error rate: the proportion of operations that fail or violate correctness criteria.
• Resource efficiency: work completed per CPU, memory, storage, or network unit.

Percentiles and tail behavior

The mean latency can conceal serious user experience problems. If 99% of requests complete in 100 ms and 1% take 10 seconds, the mean is approximately 199 ms, while the p99 is approximately 10 seconds. A user whose request falls in the slow group experiences the latter, not the average.

Percentiles must be computed from an appropriate population and time window. Aggregating averages from separate servers can hide a slow server. Histograms or suitably designed approximate quantile algorithms are generally preferable to storing only averages. Percentiles are not additive: the p99 of a complete transaction cannot normally be obtained by adding the p99 values of its individual services, because slow events may not occur together.

Little’s Law

A foundational relationship for stable systems is Little’s Law:

L is the average number of items in the system, λ is the average arrival or completion rate, and W is the average time an item spends in the system. Units must be consistent: requests = requests/second × seconds. The relationship applies to a stable long-run system in which the average number entering does not permanently exceed the average number leaving. It does not by itself predict latency or identify a bottleneck.

Worked example. An API completes 200 requests per second, and the average request latency is 0.25 seconds. The average number of in-flight requests is:

If concurrency rises to 100 while throughput remains 200 requests per second, the implied average residence time is:

The increase may indicate queueing, slower dependencies, or resource saturation.

Amdahl’s Law and optimization limits

If a fraction p of execution time can be improved by a factor s, the overall speedup is:

This assumes the workload and the non-optimized portion remain unchanged. If 80% of execution time is improved by 4×:

Optimizing the database query is valuable only if database time is a substantial part of end-to-end latency. Amdahl’s Law discourages local optimization that ignores the dominant cost.

3. Profiling and Performance Diagnosis

Profiling measures where a program spends time or memory. It should be performed with a representative workload and a clear question, such as “Which functions dominate CPU time?” or “Which allocation path causes heap growth?”

Principal profiling approaches

1. Sampling profilers periodically record the active call stack. They have relatively low overhead and are effective for identifying hot paths, but short-lived events may be missed.
2. Instrumentation profilers record entry and exit events for selected functions. They provide detailed timings but can distort behavior through measurement overhead.
3. CPU profilers identify computation-intensive methods, lock contention, and scheduling effects.
4. Memory profilers identify allocation rates, retained objects, leaks, and garbage-collection pressure.
5. I/O and system profilers reveal disk, network, system-call, and kernel waiting time.
6. Database profilers expose slow queries, inefficient plans, lock waits, and excessive round trips.

A disciplined profiling process is: establish a baseline; reproduce the workload; profile in an environment close to production; identify a bottleneck; change one significant factor; rerun the same test; and compare distributions, not just a single average. Profiling production traffic can be valuable, but privacy, overhead, and security controls are essential.

Common mistakes include optimizing code before measuring it, profiling an unrealistic microbenchmark, ignoring garbage collection or I/O wait, and treating a flame graph as proof of causation. A wide frame shows sampled time beneath a call path; it does not automatically prove that the function is independently inefficient.

4. Load Testing and Capacity Planning

Load testing evaluates behavior under controlled demand. A complete test plan specifies workload mix, arrival pattern, data volume, client behavior, environment, success criteria, warm-up period, duration, and stopping conditions.

Types of tests

• Load test: expected normal or peak demand.
• Stress test: demand beyond the expected operating point to find failure behavior.
• Spike test: rapid change in demand.
• Soak test: sustained demand to reveal leaks, degradation, or resource exhaustion.
• Scalability test: measures behavior as resources or workload increase.
• Capacity test: estimates the maximum demand meeting defined SLOs.
• Fault-injection test: introduces dependency or infrastructure failures.

A useful test must model more than request volume. It should include realistic payload sizes, cache states, authentication, user think time, read/write ratios, geographic distribution, and downstream dependencies. Synthetic traffic that always requests the same cached object can produce dangerously optimistic results.

Queueing and utilization

As utilization approaches the capacity of a bottleneck, queueing delay commonly rises nonlinearly. A server with average CPU utilization of 50% may have substantial spare capacity, but if one thread pool, lock, database connection pool, or disk queue is saturated, the service can still fail its latency objective. Therefore, measure each relevant resource and distinguish average utilization from saturation and queue length.

A simple capacity estimate is:

where D_peak is peak demand in operations per second, C_target is the sustainable capacity per instance while meeting SLOs, and ceil rounds upward. This is only a first estimate. A planning margin is usually required for failover, demand uncertainty, deployment loss, and correlated degradation.

Worked example. A service must handle 1,200 requests per second. Load testing shows that one instance sustains 250 requests per second while meeting the latency SLO. The minimum instance count is:

If one instance must be unavailable during maintenance or failure, at least six are needed for the same tested capacity, before considering additional headroom. The result is valid only if the benchmark includes the relevant workload and scaling mechanism.

Capacity planning should use demand forecasts, historical seasonality, resource limits, and failure scenarios. A forecast can be represented as a range rather than a point estimate. Teams should ask not only “How much capacity is needed?” but also “How quickly can capacity be added?” and “What happens while scaling is in progress?”

5. Designing for Fault Tolerance and Resilience

Fault tolerance does not mean that failures are impossible. It means that the system has defined behavior when components fail. Good designs identify failure domains, limit blast radius, preserve critical functions, and recover safely.

Important patterns include:

• Timeouts: prevent indefinite waiting. They must be chosen with knowledge of normal latency and downstream behavior.
• Retries: useful for transient failures, but dangerous when requests are non-idempotent or when retries amplify overload. Exponential backoff and randomized jitter reduce synchronized retry storms.
• Circuit breakers: stop calls to an unhealthy dependency for a period, allowing recovery and protecting callers.
• Bulkheads: isolate resources such as worker pools or connection pools so one workload cannot exhaust all capacity.
• Rate limiting: controls demand and protects finite resources.
• Load shedding: deliberately rejects lower-priority work to preserve essential operations.
• Caching: reduces dependency load but introduces staleness, invalidation complexity, and possible privacy risks.
• Replication and failover: improve continuity but create consistency and operational complexity.
• Idempotency keys: allow safe retry of operations such as payment requests when the same logical request may be transmitted more than once.
• Graceful degradation: provides reduced but useful functionality during partial failure.

Retries require special attention. If a caller sends r attempts for each logical request across a chain of n independent layers, the potential amplification can approach rⁿ in a worst-case retry design. This is not a universal traffic equation—real systems impose limits—but it illustrates why retries should normally be bounded, coordinated, observable, and applied only to appropriate errors.

Consistency and availability trade-offs must be explicit. A stale product catalogue may be acceptable during a replica outage, whereas a stale account balance may not be. Fallback behavior should be classified by business criticality rather than applied uniformly.

6. Observability: Logs, Metrics, and Traces

Observability is the capability to infer internal states from system outputs. It is broader than monitoring. Monitoring commonly asks whether known conditions are healthy; observability also supports investigation of unexpected conditions.

Logs

A useful log event records what happened, when, where, and with which correlation context. Structured formats are generally easier to query than free-form text. A production event may contain a timestamp, severity, service, operation, request ID, trace ID, outcome, duration, and safe diagnostic fields.

Logs should be actionable and appropriately sampled. Excessive logs increase cost and may conceal important events. Sensitive data such as passwords, access tokens, payment data, and unnecessary personal information must not be logged. Retention should reflect operational, legal, and security requirements.

Metrics

Metrics aggregate observations over time and are suitable for alerting and trend analysis. Common categories are:

• Counters: monotonically increasing totals, such as requests or errors.
• Gauges: values that rise or fall, such as queue depth or memory use.
• Histograms: distributions of observations such as request duration.
• Summaries: client- or library-calculated quantile summaries with limitations for aggregation.

The four golden signals are latency, traffic, errors, and saturation. Metrics should use dimensions that support diagnosis without creating unbounded cardinality. A label containing a unique user ID or request ID can make a metric system expensive and ineffective; those identifiers belong in traces or logs.

Distributed tracing

A trace follows one logical request across services. It consists of spans representing operations, with parent–child relationships and timing information. Traces reveal fan-out, critical paths, retries, queueing, and dependency delays that are difficult to reconstruct from independent logs.

Trace context must be propagated securely across process boundaries. Sampling reduces overhead, but tail-based or adaptive sampling may be needed to retain rare slow or failed requests. Correlation IDs should never be treated as authorization credentials.

Alert design

An alert should identify a user-impacting or imminent condition, include a clear owner, and suggest an action. Alerts based only on CPU can be noisy; an alert based on SLO burn rate may better represent customer impact. A useful alerting system distinguishes pages requiring immediate human response from tickets or dashboards intended for later investigation.

A burn-rate concept compares the current error consumption rate with the permitted rate. If a service is consuming its error budget 10 times faster than the sustainable rate, the situation is urgent even if the monthly SLO has not yet been breached. Exact thresholds depend on the service, observation window, and operational policy.

7. Chaos Engineering and Controlled Failure

Chaos engineering is the disciplined practice of experimenting on a system to learn how it behaves under realistic adverse conditions. It is not random destruction. A hypothesis should state what is expected, such as: “If one availability-zone instance is terminated, the checkout service will maintain the request-success SLO and recover capacity within five minutes.”

A safe experiment includes a defined scope, baseline, steady-state metric, fault, abort conditions, observers, permissions, and rollback plan. Experiments should begin in non-production environments and progress carefully. Production experiments require business approval, clear safeguards, and a way to stop the fault.

Possible faults include process termination, network delay, packet loss, dependency errors, expired certificates, disk exhaustion, clock anomalies, and regional loss. The experiment should test a meaningful failure mode rather than a convenient one. Results must be recorded, including unexpected effects and remediation actions.

Chaos testing can expose hidden coupling, inadequate timeouts, poor retry behavior, missing capacity, and misleading dashboards. It cannot prove that a system will survive every failure. Its value comes from repeatedly reducing uncertainty and validating operational assumptions.

8. Incident Management and Learning

An incident is an event that threatens or causes degradation of a service. Incident management coordinates detection, response, communication, mitigation, recovery, and learning.

A typical response sequence is:

1. Detect and acknowledge the issue.
2. Assess impact, affected users, severity, and likely scope.
3. Assign roles such as incident commander, operations lead, communications lead, and scribe.
4. Stabilize the service using rollback, traffic reduction, failover, or feature disablement.
5. Diagnose using timelines, metrics, logs, traces, and recent changes.
6. Communicate internally and, when appropriate, to customers.
7. Verify recovery against SLIs rather than relying on a single green dashboard.
8. Conduct a blameless review and track corrective actions.

A blameless review does not avoid accountability. It focuses on system conditions, decisions, incentives, missing safeguards, and process weaknesses rather than treating individual error as the complete explanation. A good review identifies contributing factors, what went well, what failed, and actions with owners and due dates.

Runbooks should state symptoms, checks, commands or procedures, escalation paths, rollback steps, and safety warnings. They should be tested during drills. Incident data must be handled carefully because logs and timelines may contain personal or confidential information.

9. Integrated Case Study: Checkout Reliability

The following is an illustrative case, not a report of a particular company.

Background

An online retailer operates a checkout service composed of an API gateway, checkout application, inventory service, payment provider, and order database. Its objective is to complete at least 99.9% of valid checkout requests within 2 seconds during peak periods.

Problem

During a promotional event, the p99 checkout latency rises above 8 seconds and payment errors increase. Average CPU utilization on application servers remains below 60%, so an initial hypothesis that “the servers need more CPU” is not supported by the evidence.

Analysis

The team compares traces for successful and failed requests. Slow traces show that the checkout service waits on the payment provider. Logs reveal repeated payment attempts for some requests. Metrics show that the payment connection pool is saturated, while application CPU is moderate. A recent change increased the client timeout from 2 seconds to 10 seconds and enabled three retries for timeout responses. This allowed more requests to wait and multiplied pressure on the finite connection pool.

The team also discovers that the checkout operation lacks a robust idempotency key. A retry can therefore create uncertainty about whether a payment was accepted before the client timed out.

Approach and solution

The response is staged:

1. Roll back the timeout and retry change.
2. Limit retries to explicitly transient errors and use exponential backoff with jitter.
3. Add idempotency keys for each logical checkout attempt.
4. Introduce a circuit breaker for the payment provider.
5. Reserve connection-pool capacity for high-priority checkout operations.
6. Provide a clear “payment status pending” workflow rather than blindly repeating payment requests.
7. Add an SLI for successful checkout completion within 2 seconds and another for payment-provider timeout rate.
8. Run a load test with realistic payment latency and a fault-injection test in which the provider delays responses.
9. Update dashboards, alerts, and the incident runbook.

Outcome

In a subsequent test, the service maintains the target success rate during normal provider latency and degrades predictably during provider delay. Some transactions are placed into a safe pending state, but duplicate charges are prevented. The team also learns that low CPU utilization did not imply spare end-to-end capacity: the bottleneck was connection occupancy and dependency latency.

Learning points

The case demonstrates that performance is a system property, not merely a server-resource measure. Timeouts, retries, idempotency, observability, and business semantics must be designed together. It also illustrates why tail latency and dependency traces are more informative than averages and why resilience tests should include realistic downstream faults.

10. Governance, Ethics, and Contemporary Practice

Reliability engineering affects people and organizations. A system outage may prevent access to healthcare, income, transportation, or essential communication. SLOs should therefore reflect user and societal impact, not only technical convenience.

Observability systems must follow privacy and security principles. Data collection should be purposeful, minimized, access-controlled, encrypted where appropriate, and retained only as long as justified. Debugging convenience does not justify collecting sensitive personal data. Access to production telemetry should be audited.

Automated remediation and artificial-intelligence-assisted operations can summarize incidents, detect anomalies, recommend probable causes, or generate runbook steps. These tools can reduce cognitive load, but their outputs require human verification. Training data may contain secrets or personal data, and an automated action can amplify an incorrect diagnosis. AI should operate with bounded permissions, explainable evidence, audit trails, and safe rollback.

Contemporary systems increasingly use OpenTelemetry-style vendor-neutral instrumentation, continuous profiling, eBPF-based observation, adaptive sampling, and service-mesh telemetry. These technologies improve visibility but do not replace sound SLI design, causal reasoning, capacity testing, or human judgment. More telemetry is not automatically more observability; useful signals must answer operational questions at acceptable cost.

11. Best-Practice Checklist

A professional team should:

• Define user-centered SLIs and measurable SLOs.
• Report latency distributions and error rates, not only averages.
• Profile representative workloads before optimizing.
• Load-test normal, peak, spike, and sustained demand.
• Measure bottlenecks, queues, dependency behavior, and saturation.
• Plan capacity for growth, failure, maintenance, and uncertainty.
• Bound timeouts, retries, concurrency, and resource consumption.
• Use idempotency and safe degradation for important operations.
• Instrument logs, metrics, and traces with correlation context.
• Protect telemetry from excessive cost and sensitive-data exposure.
• Practice incident response and test runbooks.
• Run controlled chaos experiments with explicit hypotheses and abort criteria.
• Conduct blameless post-incident reviews and verify corrective actions.
• Treat reliability as a product and governance responsibility, not only an operations task.

Challenges and Limitations

No single metric captures reliability or performance. Availability can hide incorrect responses; latency can hide failed requests; averages can hide tails; and dashboards can create false confidence when instrumentation is incomplete. Quantitative models often assume independence, stationarity, or stable workload, while real distributed systems exhibit correlated failures and changing demand.

Load tests can be unrepresentative, especially when they omit cache behavior, data skew, user think time, or external dependencies. Chaos experiments have limited coverage and may create risk if governance is weak. Observability introduces storage, processing, and privacy costs. SLOs can also be gamed if teams choose easy-to-measure indicators that do not represent user outcomes.

The appropriate response is not to abandon measurement, but to state assumptions, triangulate evidence, validate models with experiments, and revise instrumentation as the system and its users change.

Chapter Summary

Reliability, performance, and observability form an integrated foundation for advanced software development. Reliability requires explicit functions, operating conditions, and failure definitions. Availability can be estimated from usable and unusable time, while SLOs and error budgets connect technical behavior to service decisions.

Performance analysis uses latency distributions, throughput, concurrency, saturation, error rate, and resource efficiency. Little’s Law links concurrency, throughput, and residence time in stable systems; Amdahl’s Law shows why optimization is limited by the portion of work that cannot be improved. Profiling identifies bottlenecks, while representative load, stress, spike, soak, and capacity tests reveal behavior under demand.

Resilient systems use bounded timeouts, controlled retries, circuit breakers, bulkheads, rate limiting, load shedding, replication, idempotency, and graceful degradation. Observability combines structured logs, metrics, and distributed traces to support detection and diagnosis. Chaos engineering validates hypotheses about failure behavior, and incident management provides coordinated response and organizational learning. Ethical practice requires privacy protection, secure telemetry, transparent governance, and human oversight of automated operational tools.

Key Terms

Reliability: Probability that a system performs its required function under stated conditions for a specified time.

Availability: Proportion of time that a service is usable when required.

Resilience: Ability to withstand disruption, degrade acceptably, recover, and learn.

Performance: Characteristics of speed, capacity, and resource consumption.

Scalability: Ability to handle increased demand through additional resources or design changes.

SLI: Quantitative indicator measuring a service property.

SLO: Target value or range for an SLI over a specified period.

SLA: Formal service commitment, often contractual, with defined consequences.

Error budget: Permitted amount of unreliability implied by an SLO.

Latency: Time elapsed while completing an operation.

Throughput: Number of completed operations per unit time.

Saturation: Degree to which a resource or queue approaches its limit.

Profiling: Measurement of program execution, resource use, or call behavior.

Load testing: Controlled evaluation under a specified workload.

Capacity planning: Estimation and preparation of resources needed for demand and failure conditions.

Circuit breaker: Pattern that temporarily stops calls to an unhealthy dependency.

Idempotency: Property that repeating a logical operation has the same intended effect as performing it once.

Observability: Ability to infer internal system state from external outputs.

Distributed trace: Connected collection of spans following a request across components.

Chaos engineering: Controlled experimentation to learn system behavior under failure.

Review Questions

Short Answer Questions

1. Distinguish an SLI, an SLO, and an SLA.
2. Why can p99 latency be more useful than average latency for user-facing services?
3. State two risks associated with unrestricted retries.
4. What are the three principal observability signals discussed in this chapter?
5. Why should a chaos experiment include abort conditions?

Long Answer Questions

1. Explain how profiling, load testing, and capacity planning complement one another in performance engineering.
2. Discuss the assumptions and limitations of the series and parallel availability formulas.
3. Explain how timeouts, retries, circuit breakers, bulkheads, and idempotency contribute to resilience.
4. Describe an incident-management process from detection through post-incident learning.
5. Evaluate the ethical and governance risks of collecting production logs, metrics, and traces.

Analytical / Application Questions

1. A service has an SLO of 99.95% over a 30-day month. Calculate the approximate time-based error budget in minutes and explain why this calculation may not apply to a request-based SLO.
2. A system completes 400 requests per second with an average latency of 150 ms. Use Little’s Law to estimate average concurrency, state the unit conversion, and identify one condition required for the calculation to be meaningful.
3. A payment dependency begins timing out during a traffic spike. Propose a resilience and observability response that protects against duplicate charges, limits retry amplification, preserves high-priority work, and provides evidence for later diagnosis.

Multiple Choice Questions with Answers

Question 1

An SLO specifies that 99.9% of valid requests must succeed within 300 ms over one month. What is the SLI?

A. The contractual service-credit policy

B. The measured proportion of valid requests meeting the 300 ms success condition

C. The engineering team’s monthly error budget

D. The number of application servers

Correct Answer: B. The measured proportion of valid requests meeting the 300 ms success condition

Question 2

A service completes 200 requests per second and has an average latency of 0.25 seconds. According to Little’s Law, what is the average concurrency?

A. 50 requests

B. 80 requests

C. 200 requests

D. 800 requests

Correct Answer: A. 50 requests

Question 3

Which measurement is most likely to reveal the experience of the slowest 1% of requests?

A. Mean CPU utilization

B. Request count

C. p99 latency

D. Total log volume

Correct Answer: C. p99 latency

Question 4

What is the principal danger of unlimited retries during a dependency outage?

A. They always reduce network traffic

B. They can amplify load and worsen the outage

C. They eliminate the need for timeouts

D. They guarantee idempotent operations

Correct Answer: B. They can amplify load and worsen the outage

Question 5

Which pattern temporarily stops requests to an unhealthy dependency so that it can recover?

A. Circuit breaker

B. Cache warmer

C. Database index

D. Compiler optimizer

Correct Answer: A. Circuit breaker

Question 6

A service must handle 1,200 requests per second, and one tested instance sustains 250 requests per second while meeting its SLO. What is the minimum number of instances before adding failure headroom?

A. 4

B. 5

C. 6

D. 8

Correct Answer: B. 5

Question 7

Which observability signal is best suited to show the path and timing of one request across multiple services?

A. Gauge

B. Distributed trace

C. Availability percentage alone

D. Static configuration file

Correct Answer: B. Distributed trace

Question 8

What is a defining characteristic of a well-designed chaos experiment?

A. It has no hypothesis so results remain unlimited

B. It intentionally maximizes damage

C. It specifies expected steady state, scope, and abort conditions

D. It replaces all production monitoring

Correct Answer: C. It specifies expected steady state, scope, and abort conditions

Question 9

Why is idempotency especially important in payment processing?

A. It guarantees zero latency

B. It prevents a retried logical request from unintentionally charging multiple times

C. It removes the need for authentication

D. It increases database inconsistency

Correct Answer: B. It prevents a retried logical request from unintentionally charging multiple times

Question 10

If 80% of execution time is improved by a factor of 4, what overall speedup does Amdahl’s Law predict?

A. 1.25×

B. 2×

C. 2.5×

D. 4×

Correct Answer: C. 2.5×