Observability, Reliability and Incident ManagementIntroductionCloud applications fail in ways that are both technical and organizational. A database may become slow, a deployment may introduce an error, a third-party API may be unavailable, or a configuration change may exhaust memory. In a distributed cloud system, users often experience the failure before engineers understand its cause. Observability and reliability engineering provide the practices needed to detect, explain, prevent, and recover from such events. Heroku and AWS provide complementary capabilities. Heroku supplies managed application deployment, platform logs, release management, metrics integrations, and operational controls. AWS provides CloudWatch metrics, logs, alarms, dashboards, X-Ray tracing, CloudTrail auditing, Auto Scaling, load balancing, backups, and multiple availability options. These tools are valuable only when connected to clearly defined service objectives and disciplined operating procedures. This chapter explains how to design an observability system, define service-level indicators (SLIs), service-level objectives (SLOs), and error budgets, plan capacity, conduct root-cause analysis, design redundancy and graceful degradation, and manage incidents and disaster recovery. Learning ObjectivesBy the end of this chapter, you should be able to: - Distinguish monitoring from observability and explain the roles of logs, metrics, and traces.
- Implement practical observability using Heroku and AWS tooling.
- Define SLIs, SLOs, service-level agreements (SLAs), and error budgets.
- Design dashboards and alerts that support rapid diagnosis rather than alert noise.
- Apply capacity-planning methods and identify bottlenecks.
- Conduct incident response, root-cause analysis, and blameless postmortems.
- Design redundancy, graceful degradation, backup, recovery, RTO, and RPO strategies.
- Produce operational runbooks suitable for professional software teams.
1. Observability as an Engineering DisciplineMonitoring asks whether a known condition is occurring: for example, whether CPU utilization exceeds 80%. Observability asks whether the internal state of a system can be inferred from its external outputs. A system is more observable when engineers can explain an unfamiliar failure using available evidence without modifying the system first. The traditional three pillars are: - Logs: timestamped records of discrete events.
- Metrics: numerical measurements aggregated over time.
- Traces: records of a request’s journey across services.
These pillars complement one another. A metric may show that latency increased; a trace may identify the slow service; and a log may explain that a database connection pool was exhausted. 1.1 LogsA useful production log is structured, searchable, and contextual. JSON logs are generally preferable to unstructured sentences because fields such as request_id, service, environment, status_code, and duration_ms can be filtered and aggregated. A log event should normally include: - UTC timestamp;
- severity, such as DEBUG, INFO, WARN, or ERROR;
- application and environment;
- correlation or request identifier;
- event name;
- relevant operational fields;
- an error type and stack trace when appropriate.
Logs must not contain passwords, access tokens, payment-card data, or unnecessary personal information. Heroku applications write to standard output and standard error, which are collected by the Heroku Logplex system and exposed through the Heroku CLI, dashboard, and log drains. A production application commonly sends logs to a managed analysis service through a drain. AWS applications can send logs to CloudWatch Logs, where log groups, retention policies, metric filters, and queries support analysis. Log levels should have operational meaning. Excessive DEBUG logging increases cost and can obscure important events; insufficient logging makes diagnosis impossible. Logging every request body, for example, may create privacy and storage risks. A better design logs the request type, identity surrogate, outcome, and latency while excluding sensitive payloads. 1.2 MetricsMetrics are efficient for detecting trends and triggering alerts. Common categories include: | Category | Examples | Main use |
|---|
| Traffic | Requests per second, jobs processed | Demand and capacity | | Errors | 5xx rate, failed jobs | Correctness and availability | | Latency | p50, p95, p99 response time | User experience | | Saturation | CPU, memory, queue depth, connections | Resource exhaustion | | Business | Checkouts, uploads, registrations | User and organizational impact |
The four golden signals are latency, traffic, errors, and saturation. They should be measured at meaningful boundaries, such as the application endpoint or user transaction, rather than inferred only from infrastructure utilization. Percentiles are particularly useful. The p95 latency is the value below which 95% of observations fall. It does not mean that 95% of requests take exactly that time, and it does not describe the worst request. Percentiles should be calculated over suitable windows and with adequate sample sizes. 1.3 TracesA distributed trace follows one transaction through multiple components. A trace contains spans, and each span records an operation, start and end time, attributes, and status. A trace context propagated through HTTP headers or messaging metadata allows a request in a Heroku web process to be associated with database, cache, and AWS service operations. OpenTelemetry is a widely adopted instrumentation standard. It can export traces, metrics, and logs to compatible back ends. AWS X-Ray supports tracing for many AWS services and can be integrated with instrumented applications. Heroku does not provide a complete native distributed-tracing platform, so applications commonly use OpenTelemetry libraries and an external collector or observability provider. Traces are expensive if every request is retained indefinitely. Sampling can reduce volume, but error traces and slow traces should receive higher priority. Sampling decisions must preserve enough evidence for statistically meaningful diagnosis. 2. Instrumentation and Tooling with Heroku and AWSObservability begins with consistent instrumentation, not with dashboards. Every service should define a small vocabulary of events and dimensions. Useful dimensions include endpoint, HTTP method, status class, release version, region, and dependency. High-cardinality fields, such as unrestricted user IDs or URLs containing unique tokens, can make metrics expensive and difficult to query. 2.1 Heroku practicesA practical Heroku observability design includes: - Structured application logs written to stdout or stderr.
- Heroku router and application log streams collected through the CLI or a log drain.
- Release identifiers included in logs and deployment metadata.
- Application metrics exported to a monitoring provider or collected through an application endpoint.
- Dyno and database health monitored through Heroku’s dashboard and supported add-ons.
- Deployment and rollback procedures documented in a runbook.
Heroku’s ephemeral filesystem should not be treated as durable storage. Files required after a dyno restart belong in a database or object store such as Amazon S3. A dyno restart, scaling action, or platform event should therefore be considered a normal operational condition rather than an exceptional backup mechanism. 2.2 AWS practicesA typical AWS observability stack may use: - CloudWatch Metrics for infrastructure and application measurements;
- CloudWatch Logs for centralized log storage and queries;
- CloudWatch Alarms for threshold or anomaly-based notifications;
- CloudWatch Dashboards for operational views;
- AWS X-Ray or OpenTelemetry-compatible tools for traces;
- CloudTrail for account and API activity auditing;
- AWS Config and security services for configuration governance.
CloudWatch alarms should be connected to an action such as an Amazon SNS notification, an incident-management platform, or an Auto Scaling policy. A dashboard is not itself an alerting system. Engineers should know which signals are actionable, who receives them, and what the first response should be. 2.3 Correlation and contextA request identifier should be generated at the system boundary if one is absent and propagated across services. Logs, traces, and error reports should use the same identifier. Release version, deployment identifier, and configuration version are also important because many incidents begin immediately after a change. A useful event might conceptually contain: {timestamp, level, service, environment, release, request_id, trace_id, route, status, duration_ms, error_type}
The exact schema may vary, but consistency is more important than any particular field name. 3. SLIs, SLOs, SLAs and Error BudgetsAn SLI is a quantitative measure of a service aspect. An SLO is a target for that measure over a specified period. An SLA is a business or contractual commitment, often including remedies if the commitment is missed. These terms should not be used interchangeably. 3.1 Defining an SLIFor request availability, a useful event-based SLI is: Availability = successful eligible requests ÷ total eligible requests A successful request must be defined. For example, HTTP 2xx and selected 3xx responses may count as successful, while server errors, timeouts, and connection failures do not. Health checks, deliberately rejected invalid requests, and bot traffic may need separate treatment. The eligibility rule must be explicit and resistant to manipulation. For latency, an SLI may be the proportion of eligible requests completed within a threshold T: Latency compliance = requests with latency ≤ T ÷ total eligible requests The numerator and denominator must refer to the same population and time window. A latency target should be expressed with a percentile or compliance objective, such as “99% of checkout requests complete within 800 ms over a rolling 30-day period.” 3.2 SLOs and error budgetsIf an availability SLO is 99.9% over a 30-day period, the permitted unavailability is: Error budget = 1 − SLO Thus, the error budget is 0.1% of eligible service time or requests, depending on the SLI definition. If measured by time, using a simplified 30-day month: 30 × 24 × 60 = 43,200 minutes 43,200 × 0.001 = 43.2 minutes The service may therefore be unavailable for approximately 43.2 minutes in that assumed month while remaining within the target. The calculation is an approximation because calendar months have different lengths and because request-based and time-based SLIs are not equivalent. An error budget creates a decision rule. When the budget is healthy, a team may prioritize feature delivery and controlled experimentation. When the budget is being consumed rapidly, the team should prioritize reliability work, reduce risky changes, or impose a deployment freeze. This is not a punishment mechanism; it is a way to balance innovation and reliability using evidence. 3.3 Good objectivesAn SLO should be: - user-centered;
- measurable from a defined event population;
- time-bounded;
- realistic but demanding;
- independent of misleading implementation details;
- connected to an operational decision.
“Keep CPU below 70%” is usually a poor user-facing SLO. CPU may be high while users are satisfied, or low while a dependency is failing. “99.5% of authenticated API requests return a valid response within 500 ms” is closer to a useful objective, although it still requires careful eligibility and measurement definitions. 4. Dashboards and Alert DesignA dashboard should answer a specific operational question. A service overview commonly contains request volume, error rate, latency percentiles, saturation, dependency health, active incidents, and recent releases. A capacity dashboard may additionally show CPU, memory, database connections, queue depth, storage growth, and autoscaling activity. Dashboards should show time comparisons and sufficient context. A single red number without a baseline is difficult to interpret. Annotations for deployments, configuration changes, and incidents make correlations visible. 4.1 Alert principlesAn alert is useful when it is actionable, urgent enough to interrupt work, and assigned to a responder. Alerts should generally be based on symptoms that users experience, not on every unusual internal measurement. Important alert properties include: - Threshold: the condition that activates the alert;
- duration: how long the condition must persist;
- severity: the expected impact and response time;
- owner: the responsible team or rotation;
- runbook link: the first diagnostic and mitigation steps;
- deduplication: prevention of one incident producing hundreds of notifications.
A multi-window burn-rate alert can detect rapid SLO consumption. The exact policy depends on the organization, but the principle is to alert when the recent rate of error-budget consumption threatens the longer-term objective. A low-volume service requires special care because a few failures can produce unstable percentages. Alert fatigue is a reliability risk. If responders routinely ignore alerts, the system is effectively unmonitored. Review alerts after incidents and remove alerts that do not lead to action. 5. Reliability ArchitectureReliability is the ability of a system to perform its required function over time and under stated conditions. It is not identical to availability. A service may be available but return incorrect results; it may be reliable for ordinary traffic but fail under a predictable peak. 5.1 Redundancy and failure domainsRedundancy means providing more than one component or path so that failure of one does not stop the service. Examples include multiple Heroku dynos, multiple application instances behind an AWS load balancer, database replicas, multi-AZ deployment, and replicated object storage. Redundancy is effective only when failure domains are independent. Two instances in the same failed availability zone, or two services depending on one unreplicated database, may provide little practical resilience. Shared dependencies, credentials, DNS, deployment pipelines, and network paths must be considered. 5.2 Graceful degradationGraceful degradation preserves essential functions when optional functions fail. Examples include: - serving cached product information when recommendations are unavailable;
- accepting an order while delaying a nonessential email;
- returning a partial response rather than failing the entire page;
- using a queue to absorb temporary downstream unavailability;
- applying timeouts, retries with bounded backoff, and circuit breakers.
Retries must be designed carefully. Retrying a non-idempotent payment operation can create duplicate charges. Exponential backoff with jitter reduces synchronized retry storms, but it does not solve a permanently failing dependency. Timeouts prevent resources from remaining occupied indefinitely. 5.3 Health checks and deployment safetyA liveness check asks whether a process should be restarted; a readiness check asks whether it should receive traffic. A readiness check that tests every dependency can cause cascading removal of all instances during a dependency outage. Checks should therefore reflect whether the service can safely serve its essential contract. Safe deployment practices include small changes, automated tests, migration compatibility, canary or phased releases where supported, monitoring after release, and rapid rollback. Database migrations deserve particular care because code rollback does not automatically reverse an incompatible schema change. 6. Capacity PlanningCapacity planning connects demand forecasts to resource requirements. It is not limited to CPU. Memory, database connections, storage, network bandwidth, queue workers, API quotas, and rate limits may become the binding constraint. A basic utilization relationship is: Utilization ρ = offered workload ÷ sustainable capacity Here, ρ is dimensionless. The workload and capacity must be measured in the same units, such as requests per second. The relationship is a planning approximation; real systems become nonlinear as queues grow and dependencies saturate. For a simple stateless service, if peak demand is 240 requests per second and one instance safely handles 80 requests per second, the theoretical minimum is: Nmin = 240 ÷ 80 = 3 instances If the design requires one instance to be unavailable while maintaining peak capacity, at least four instances are needed. In practice, engineers add headroom for bursts, measurement uncertainty, deployment capacity, and dependency limits. The calculation must be validated with load testing because throughput depends on request mix, latency target, memory, and downstream services. Queue systems require additional analysis. If arrivals temporarily exceed processing capacity, queue depth increases. Monitoring queue age is often more meaningful than queue length because a queue of 1,000 small jobs may be less urgent than a queue of 20 jobs containing long-running tasks. Capacity plans should record assumptions, forecast horizons, scaling triggers, maximum limits, cost implications, and the procedure for responding when the forecast is wrong. 7. Incident ManagementAn incident is an unplanned event that reduces or threatens service quality. Incident management is an organized process for restoring service and learning from failure. A practical incident lifecycle is: - Detection: an alert, customer report, or engineer identifies a problem.
- Triage: responders establish impact, scope, severity, and start time.
- Containment: the team prevents further harm, possibly by disabling a feature or stopping a deployment.
- Mitigation: a temporary action restores acceptable service.
- Recovery: normal operation is restored and data correctness is checked.
- Communication: stakeholders receive timely, accurate updates.
- Learning: evidence is preserved and follow-up work is assigned.
Incident roles may include incident commander, operations lead, communications lead, and subject-matter experts. Separating coordination from deep technical investigation reduces confusion. During a major event, the incident commander should maintain a shared timeline and make explicit decisions about priorities. 7.1 Root-cause analysisRoot-cause analysis should explain both the immediate technical mechanism and the conditions that allowed it to occur. A useful investigation gathers logs, metrics, traces, deployment records, configuration changes, dependency status, and user reports. Correlation is not proof: a deployment occurring before an outage is evidence to investigate, not conclusive proof of causation. Methods include the “five whys,” fault-tree analysis, causal timelines, and contributing-factor analysis. A mature analysis may identify several causes: an unsafe migration, an inadequate timeout, missing capacity headroom, and an alert that was not routed to the correct team. The goal is not to identify a person to blame. Blame reduces reporting and hides systemic weaknesses. Accountability still matters: owners, deadlines, and verification criteria must be assigned for corrective actions. 8. Runbooks, Postmortems and Operational GovernanceA runbook is an executable operational guide. It should state the purpose, symptoms, prerequisites, diagnostic commands or console paths, decision points, mitigation steps, rollback instructions, escalation contacts, validation checks, and recovery actions. Commands should identify whether they are read-only or potentially destructive. Secrets should never be embedded in runbooks. A postmortem should include: - incident title, severity, and duration;
- affected services and users;
- concise impact statement;
- timeline with timestamps and sources;
- detection and response assessment;
- technical and organizational contributing factors;
- what worked and what failed;
- corrective actions with owners and due dates;
- evidence that actions are complete.
A postmortem is not complete when the document is written. It is complete when high-value actions have reduced recurrence risk and the organization has verified the improvement. Operational governance also includes access control, change approval proportional to risk, audit logging, retention policies, and separation of duties. CloudTrail can help identify who changed an AWS resource and when. Heroku deployment and release records similarly support change reconstruction. Observability data itself may contain personal or confidential information, so access and retention must follow applicable privacy and security requirements. 9. Disaster Recovery: RTO and RPODisaster recovery addresses events that exceed ordinary incident mitigation, such as regional outages, destructive data errors, credential compromise, or loss of a critical service. - Recovery Time Objective (RTO): the maximum acceptable time to restore the service after a disruptive event.
- Recovery Point Objective (RPO): the maximum acceptable amount of data loss measured in time.
An RPO of 15 minutes means that, under the recovery design, the organization accepts losing at most approximately 15 minutes of recently changed data. It does not mean recovery will always occur within 15 minutes. RTO and RPO are requirements that influence architecture, backup frequency, replication, cost, and testing. Common recovery patterns include: | Pattern | Recovery speed | Cost and complexity | Typical characteristic |
|---|
| Backup and restore | Slowest | Lower | Standby infrastructure created during recovery | | Pilot light | Moderate | Moderate | Core data and minimal services kept ready | | Warm standby | Fast | Higher | Reduced-capacity environment running continuously | | Active-active | Fastest potential | Highest | Multiple environments serve production traffic |
Backups must be encrypted, access-controlled, versioned where appropriate, and tested through restoration. Replication is not a substitute for backups: a corrupted or deleted record may be replicated immediately to every replica. Recovery exercises should verify application configuration, secrets, DNS, dependencies, database consistency, and staff procedures—not only whether a server can start. 10. Illustrative Case: Checkout Failure After a ReleaseBackgroundA fictional online retailer operates a Node.js API on Heroku, uses PostgreSQL for orders, Redis for short-lived data, and an AWS-hosted payment integration. The team has defined an objective that 99.5% of eligible checkout requests complete successfully over a rolling 30-day period. ProblemTen minutes after a release, the checkout error rate rises from 0.3% to 8%. Customers report intermittent failures. CPU is normal, so an infrastructure-only dashboard initially suggests no problem. AnalysisThe service dashboard shows increased checkout latency and HTTP 502 responses. Traces reveal that the payment dependency is called twice for some requests. Logs show a new retry path. The retry occurs after a timeout that is shorter than the payment provider’s processing time. Because the operation is not safely idempotent in all cases, some requests also require manual payment reconciliation. The release annotation on the dashboard aligns with the beginning of the incident. Cloud and application evidence therefore support the release as a contributing factor, but the deeper design problem is the absence of a robust idempotency key and an unsafe retry policy. Approach and solutionThe incident commander stops further deployment and rolls back the application code. The team disables automatic retries for payment calls, increases diagnostic logging without recording sensitive payment data, and verifies payment status through the provider’s reconciliation mechanism. A feature flag temporarily disables a nonessential promotional calculation, reducing checkout work. After service stabilizes, the team implements provider-supported idempotency keys, bounded timeouts, explicit payment state transitions, and a reconciliation job. A new trace spans the payment attempt and records whether the result is confirmed, pending, or failed. An SLO burn-rate alert is added for checkout errors, and the runbook is updated with rollback and payment-reconciliation steps. OutcomeThe immediate outage lasts 24 minutes. The team restores successful checkout, identifies potentially affected transactions, and contacts customers where necessary. A postmortem assigns owners for code changes, alert improvement, a payment failure-mode test, and a disaster-recovery exercise. Learning points- Normal CPU does not imply healthy user transactions.
- Release annotations and correlation identifiers shorten diagnosis.
- Retries require idempotency and bounded behavior.
- Graceful degradation can preserve core checkout when optional work fails.
- Recovery includes data and payment correctness, not merely returning HTTP 200.
11. Contemporary Developments and LimitationsOpenTelemetry is encouraging common instrumentation across vendors, reducing dependence on a single observability format. eBPF-based tools can provide kernel and network visibility with limited application changes, although they require suitable expertise and careful privacy controls. Machine-learning-assisted anomaly detection can identify unusual patterns, but it should supplement—not replace—well-defined SLOs and human judgment. AI-generated incident summaries may accelerate triage, but engineers must verify summaries against primary evidence and avoid sending confidential logs to unauthorized systems. Observability also has limitations. Metrics can hide rare failures through aggregation; logs can be incomplete or misleading; traces can be sampled; and dashboards can create false confidence. More telemetry is not always better. Instrumentation has cost, performance, privacy, and retention implications. Reliability engineering therefore involves selecting evidence that supports decisions, not collecting everything indiscriminately. Best-Practice Checklist- Define user-centered SLIs before configuring alerts.
- Use structured logs with correlation and release identifiers.
- Measure latency distributions, not only averages.
- Centralize Heroku and AWS evidence where appropriate.
- Alert on actionable symptoms and link every alert to a runbook.
- Test redundancy by considering shared failure domains.
- Use timeouts, bounded retries, idempotency, and circuit breaking.
- Plan capacity against peak demand and the loss of a component.
- Practice backup restoration and disaster recovery.
- Record RTO and RPO as explicit business requirements.
- Conduct blameless postmortems with owned corrective actions.
- Review telemetry cost, privacy, retention, and access regularly.
Chapter SummaryObservability enables engineers to infer system state from logs, metrics, and traces. Heroku applications can expose structured logs through platform collection and drains, while AWS supplies CloudWatch, X-Ray, CloudTrail, alarms, and dashboards. Effective observability depends on consistent instrumentation, correlation identifiers, release context, and careful control of sensitive data. SLIs measure service behavior, SLOs establish targets, and SLAs express contractual commitments. Error budgets translate reliability objectives into practical decisions about releases and engineering priorities. Dashboards should provide operational context, while alerts should be actionable and owned. Reliable cloud systems use independent redundancy, safe deployment, timeouts, controlled retries, idempotency, queues, and graceful degradation. Capacity planning evaluates demand, sustainable capacity, saturation, and failure headroom. Incident management progresses from detection through mitigation, recovery, communication, and learning. Root-cause analysis examines technical and systemic contributing factors without assigning blame. Runbooks make response repeatable; postmortems convert incidents into organizational learning. Disaster recovery depends on tested backups, replication, recovery procedures, and explicit RTO and RPO requirements. Together, these practices make cloud applications more understandable, recoverable, and trustworthy. Key TermsObservability: The ability to infer internal system state from externally produced evidence. Monitoring: The collection and evaluation of known system conditions. Log: A timestamped record of an event. Metric: A numerical measurement aggregated over time. Trace: A record of a request’s path through distributed components. Span: A timed operation within a distributed trace. SLI: A quantitative service-level indicator. SLO: A target for an SLI over a defined period. SLA: A contractual service commitment. Error budget: The permitted unreliability implied by an SLO. Golden signals: Latency, traffic, errors, and saturation. Alert fatigue: Reduced responsiveness caused by excessive or irrelevant alerts. Redundancy: Additional components or paths that tolerate component failure. Graceful degradation: Preservation of essential service when optional capabilities fail. Runbook: A procedural guide for diagnosing and responding to an operational condition. Root-cause analysis: A structured investigation of causal and contributing factors. RTO: Maximum acceptable time to restore service. RPO: Maximum acceptable period of data loss. Postmortem: A documented analysis of an incident and its corrective actions. Review QuestionsShort Answer Questions- What is the difference between monitoring and observability?
- Why are correlation identifiers useful in distributed applications?
- Distinguish an SLI, an SLO, and an SLA.
- What is the purpose of an error budget?
- Define RTO and RPO.
Long Answer Questions- Explain how logs, metrics, and traces complement one another during incident diagnosis.
- Describe a practical observability architecture for an application deployed on Heroku with AWS dependencies.
- Explain how redundancy, timeouts, retries, and graceful degradation contribute to reliability.
- Discuss the design of effective dashboards and alerts, including the problem of alert fatigue.
- Compare backup-and-restore, pilot-light, warm-standby, and active-active disaster-recovery strategies.
Analytical / Application Questions- A service receives 180 requests per second at peak, and one application instance safely handles 60 requests per second. How many instances are theoretically required, and how would your answer change if one instance must be able to fail without reducing peak capacity?
- Design an SLI and SLO for a university course-registration API. Define eligible requests, success criteria, measurement window, and an operational response when the error budget is rapidly consumed.
- After a deployment, checkout failures increase while CPU and memory remain normal. Propose an incident investigation using logs, metrics, traces, deployment records, and a runbook, and identify two safe mitigation actions.
Multiple Choice Questions with AnswersQuestion 1Which observability signal is most directly suited to following one request across several services? A. Metric B. Trace C. Log-retention policy D. Capacity forecast Correct Answer: B. Trace Question 2An availability SLO is 99.9%. What fraction represents the corresponding error budget? A. 0.001 B. 0.009 C. 0.099 D. 0.999 Correct Answer: A. 0.001 Question 3Which field is most useful for connecting logs from different services handling the same request? A. Operating-system version B. Request or trace identifier C. CPU temperature D. Dashboard title Correct Answer: B. Request or trace identifier Question 4Why can replication fail to replace backups? A. Replication always reduces availability B. Replicas cannot store structured data C. Corruption or deletion may be copied to replicas D. Backups cannot be encrypted Correct Answer: C. Corruption or deletion may be copied to replicas Question 5Which alert is generally most actionable? A. A user-impacting error-budget burn condition with an owner and runbook B. Every individual debug log C. Any CPU variation of one percent D. A dashboard viewed fewer than ten times Correct Answer: A. A user-impacting error-budget burn condition with an owner and runbook Question 6A service needs 240 requests per second, and one instance safely handles 80 requests per second. What is the theoretical minimum number of instances? A. 2 B. 3 C. 4 D. 8 Correct Answer: B. 3 Question 7Which practice is essential before safely retrying a payment operation? A. Removing all timeouts B. Using an idempotency mechanism C. Increasing log verbosity indefinitely D. Disabling reconciliation Correct Answer: B. Using an idempotency mechanism Question 8What is the main purpose of a readiness check? A. To decide whether an instance should receive traffic B. To encrypt application logs C. To calculate the monthly error budget D. To replace database backups Correct Answer: A. To decide whether an instance should receive traffic Question 9What does RPO describe? A. The maximum acceptable service restoration time B. The maximum acceptable amount of data loss measured in time C. The percentage of successful requests D. The number of redundant instances Correct Answer: B. The maximum acceptable amount of data loss measured in time Question 10Which statement best describes a blameless postmortem? A. It avoids assigning responsibility for corrective actions B. It focuses on system factors while assigning owners for improvements C. It records only customer complaints D. It is completed when the incident document is drafted Correct Answer: B. It focuses on system factors while assigning owners for improvements
|