13. Observability, SRE and Production Troubleshooting

Introduction and Context

Containerized applications and Kubernetes platforms make deployment repeatable, but they also distribute application behaviour across containers, nodes, services, controllers, networks, storage systems, and external dependencies. A production failure may therefore appear in one component while its cause exists elsewhere. A pod may be reported as healthy while requests time out because a downstream database is overloaded; a deployment may succeed while a service remains unreachable because of an incorrect selector; and a node may remain operational while resource pressure causes intermittent eviction.

Observability provides the evidence required to understand such systems. It combines metrics, logs, traces, events, profiles, and contextual metadata so that engineers can infer internal system states from externally collected outputs. Site Reliability Engineering (SRE) adds an operational discipline: reliability is defined through measurable objectives, incidents are managed systematically, and engineering effort is allocated according to user impact and risk. Production troubleshooting connects these ideas through a structured process of detection, diagnosis, mitigation, recovery, and learning.

This chapter examines observability architecture in Kubernetes environments, the use of Prometheus, Grafana, and OpenTelemetry, the design of Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets, and the methods used to investigate incidents without relying on guesswork.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish monitoring from observability and explain the roles of metrics, logs, traces, events, and profiles.

2. Design an observability architecture for containerized and Kubernetes-based systems.

3. Define meaningful SLIs, SLOs, service-level agreements, and error budgets.

4. Use Prometheus, Grafana, and OpenTelemetry appropriately.

5. Apply a systematic incident-response and troubleshooting method.

6. Interpret correlations among resource, application, and user-experience signals.

7. Evaluate observability costs, security risks, data-quality problems, and governance requirements.

8. Conduct effective post-incident reviews and convert findings into reliability improvements.

From Monitoring to Observability

Monitoring traditionally asks whether known conditions are occurring: Is CPU utilization above a threshold? Is a pod restarting? Is the error rate greater than five percent? These questions remain valuable, particularly for alerting. Observability asks a broader question: given the outputs available from a system, can engineers explain why an unfamiliar behaviour is occurring?

The distinction is not absolute. Monitoring is an operational practice, whereas observability is a property of a system and its telemetry design. A system with high observability exposes sufficient, well-contextualized evidence to support diagnosis. A dashboard with hundreds of metrics may still provide poor observability if the metrics are ambiguous, disconnected from user impact, or impossible to relate to a particular request.

The principal telemetry types are complementary:

• Metrics are numerical measurements collected over time. They are efficient for alerting, capacity analysis, and trend detection.
• Logs are timestamped records of events or application statements. They provide detailed context but can be expensive and difficult to aggregate.
• Traces represent the path of a request through distributed services. They reveal latency contributions, dependency relationships, and propagation of errors.
• Events describe state changes generated by Kubernetes or other control-plane components, such as scheduling failures, image-pull errors, and container restarts.
• Profiles show where CPU, memory, locks, or other execution resources are consumed. They are especially useful for performance investigations.

Effective observability links these signals. A metric alert should lead to a dashboard; a dashboard should permit navigation to affected workloads; logs and traces should contain identifiers that allow an engineer to follow one transaction across services. Kubernetes metadata—namespace, workload, pod, node, container, deployment version, and cluster—provides essential context.

An Observability Architecture for Kubernetes

A production observability system normally has four layers. The first is instrumentation. Applications expose metrics, emit structured logs, and create traces. Kubernetes and infrastructure components expose platform telemetry. The second is collection, where agents, exporters, or gateways gather and enrich data. The third is storage and analysis, including time-series databases, log stores, and trace backends. The fourth is presentation and response through dashboards, alerts, incident tools, and operational procedures.

Prometheus is commonly used for metrics. It stores labelled time series and usually collects data by scraping HTTP endpoints. A metric such as `http_requests_total` may be labelled by service, method, route, status class, and deployment version. Prometheus Query Language (PromQL) supports aggregation, rates, ratios, and alert expressions. Exporters expose metrics for systems that do not natively provide Prometheus endpoints, while Kubernetes integrations expose information about objects, nodes, containers, and workloads.

Labels are powerful but require discipline. A label such as `status_code` generally has bounded cardinality; a label containing a user ID, request ID, or unrestricted URL can create millions of time series. High cardinality increases memory use, query cost, and operational complexity. Dynamic values should normally be retained in logs or traces rather than metric labels.

Grafana provides dashboards and visual exploration across metrics and, through integrations, logs and traces. A useful dashboard follows the relationship between user experience and causes. It may begin with request rate, error rate, and latency; continue to saturation indicators such as CPU throttling, memory pressure, queue depth, and database connections; and then provide links to workload-level logs and traces. Dashboards should be designed for a specific operational decision rather than as inventories of every available measurement.

OpenTelemetry is a vendor-neutral framework for generating, collecting, and exporting telemetry. Its APIs and SDKs support traces, metrics, and logs, while the OpenTelemetry Collector receives, processes, samples, enriches, and forwards data to selected backends. A collector can remove sensitive fields, add Kubernetes resource attributes, batch data, retry temporary failures, and route high-value traces to long-term storage. This separation reduces application dependence on a particular observability vendor.

A typical request path is as follows: an ingress or gateway receives a request; a trace context is created or continued; the request passes through several services; each service emits spans and structured logs; metrics are scraped independently; and the collector exports traces and selected logs to their backends. Correlation depends on consistent propagation of trace identifiers and consistent resource naming. Without that discipline, the organisation owns multiple data sets but cannot use them together.

Designing Useful Telemetry

Instrumentation should begin with operational questions. For an API, engineers may need to know: Are users receiving successful responses? Which routes are slow? Is a particular dependency responsible? Did the problem begin after a release? Is the failure isolated to a region, tenant, node, or version? Each question implies different measurements.

Metrics should represent rates, durations, errors, and saturation. Histograms are generally more useful than averages for latency because they permit percentile estimates and reveal tail behaviour. A mean latency of 100 milliseconds may conceal a p99 latency of several seconds. Percentiles must be interpreted carefully: a p99 over a short interval can be statistically unstable when traffic is low, and a service-wide percentile can hide a failure affecting one important route.

Structured logs should use a consistent schema containing timestamp, severity, service, environment, operation, outcome, and relevant identifiers. They should describe events rather than reproduce entire application objects. Logs must not expose passwords, access tokens, payment data, or unnecessary personal information. Sampling, retention policies, and severity standards are necessary because unrestricted logging can increase both cost and privacy risk.

Traces are most valuable for distributed requests and asynchronous workflows. Span names should be stable and meaningful, such as `GET /orders/{id}` rather than including a unique identifier. Attributes should explain important dimensions without introducing unbounded cardinality or sensitive information. Head sampling decides whether to record a trace at its start; tail sampling can retain traces that contain errors or unusually high latency after the complete trace is known. Sampling improves affordability but can obscure rare events if designed poorly.

Kubernetes events are short-lived and should not be treated as a complete audit history. They are useful during diagnosis—for example, when a pod cannot be scheduled because of insufficient resources, a volume cannot mount, or an image cannot be pulled. Audit logs, where enabled and appropriately protected, provide a more durable record of API actions and are relevant to security and governance investigations.

Reliability as a Measurable Service Property

SRE treats reliability as a user-facing outcome rather than a collection of infrastructure statistics. A service can have low CPU utilization and still be unreliable if users receive errors. Conversely, high CPU may be acceptable if latency and success objectives are met. Reliability measures therefore begin with a service description and a defined user journey.

A Service Level Indicator (SLI) is a quantitative measure of service behaviour. Common examples include the proportion of valid requests that succeed, the proportion completed within a latency threshold, freshness of a data pipeline, or availability of a scheduled workflow. An SLI should specify the event population, the measurement window, and exclusions. “Availability” is incomplete unless it states what counts as a request and what constitutes success.

A Service Level Objective (SLO) is a target for an SLI over a period, such as 99.9 percent successful requests during a rolling 30-day window. An SLO is an engineering objective, not necessarily a contractual promise. A Service Level Agreement (SLA) is an externally communicated commitment that may include remedies when targets are missed.

The error rate associated with a 99.9 percent availability objective is 0.1 percent. This permitted unreliability is the error budget. For a 30-day period, the budget represents the amount of failed or unacceptable service that can be tolerated before the objective is missed. Error budgets support balanced decisions: teams may release new features while the budget is healthy, but should prioritize reliability work when repeated incidents consume it.

Error budgets are not permission to cause failures. They are a decision framework. They should be allocated to a clearly defined service and measured consistently. Multiple SLOs may be needed because availability alone does not capture latency, correctness, durability, or freshness. At the same time, too many objectives create administrative burden and encourage teams to optimize measurements rather than user experience.

Alerting should be based primarily on symptoms and SLO impact. A page is appropriate when human action is required soon to prevent significant user harm. A warning or ticket is more appropriate for conditions that require attention but do not demand immediate response. “CPU above 80 percent” is often a weak paging rule: it may produce noise when the service is healthy and miss incidents when CPU is normal. A multi-window, multi-burn-rate alert can identify when an error budget is being consumed rapidly while reducing alerts caused by brief fluctuations.

A Systematic Incident-Response Method

Troubleshooting is an evidence-based process, not a search for a familiar command. The first step is to establish the impact: which users, regions, operations, and versions are affected? Determine when the problem began, whether it is ongoing, and whether it is worsening. Record observations in a shared incident timeline.

The next step is to stabilize the service. Possible actions include rolling back a release, disabling a feature flag, reducing traffic, increasing capacity, restarting a failed component, failing over to a healthy region, or temporarily bypassing a dependency. Mitigation should be reversible where possible and accompanied by an explicit hypothesis. A restart that restores service may be useful, but it does not establish root cause and can destroy evidence.

A practical diagnostic sequence moves from the outside inward:

1. Verify user symptoms through SLO dashboards, synthetic checks, and representative requests.

2. Establish scope by comparing services, endpoints, regions, tenants, versions, and time periods.

3. Check recent changes, including application releases, configuration, infrastructure, dependencies, certificates, and feature flags.

4. Examine golden signals: traffic, errors, latency, and saturation.

5. Trace representative failed and slow requests through their dependencies.

6. Inspect logs and Kubernetes events for the affected workload and time interval.

7. Test hypotheses with controlled comparisons rather than changing several variables at once.

8. Apply and verify mitigation, then monitor for recurrence.

The four golden signals are useful but not sufficient. A Kubernetes investigation may also examine pod readiness and liveness behaviour, restart counts, termination reasons, scheduling events, node conditions, resource requests and limits, CPU throttling, memory working set, network errors, service endpoints, ingress configuration, DNS, persistent-volume attachment, and controller reconciliation. The correct question is not “Which command should be run?” but “Which observation would distinguish these competing explanations?”

Incident command improves coordination. An incident commander manages priorities and decisions; a technical lead directs diagnosis; a communications lead updates stakeholders; and a scribe records actions and evidence. Separating these roles prevents the person performing a risky change from also having to coordinate every communication channel. Communication should state impact, current mitigation, uncertainty, and the next update time without speculating beyond evidence.

An Illustrative Case: Intermittent Checkout Failures

Background

Consider an illustrative online retailer running its checkout system on Kubernetes. The system contains an ingress gateway, checkout API, payment service, inventory service, and relational database. Traces are collected with OpenTelemetry, metrics with Prometheus, and dashboards and alerts are presented in Grafana.

Situation and Problem

Customers begin reporting that checkout sometimes fails or takes more than ten seconds. The checkout availability SLO is 99.9 percent, and the alerting system reports rapid error-budget consumption. The checkout API itself shows moderate CPU utilization, so an initial assumption that the API needs more compute appears unconvincing.

Analysis

The incident responder first confirms that the failures are real by comparing synthetic checks with customer-facing request metrics. The error ratio has increased only for checkout requests. A deployment dashboard shows that the problem began shortly after a payment-service release. Traces for failed requests show long spans in the payment service, followed by timeout errors in the checkout API. Payment-service logs contain connection-pool timeout messages.

Prometheus metrics reveal that database connection utilization in the payment service is near its configured maximum, while request concurrency has increased. Kubernetes events show no scheduling or image-pull problem, and pod restarts are normal. Comparing old and new payment-service versions shows that the new version performs an additional database query for each payment attempt. The query is valid but increases connection occupancy and duration under production concurrency.

Approach and Solution

The incident commander disables the new payment feature path using a feature flag and routes payment requests to the previous implementation. Error rates and latency decline within minutes. The team then verifies that database connections recover and that successful checkout transactions return to their normal level. The release is rolled back after preserving relevant traces, logs, query measurements, and configuration information.

During the follow-up, engineers optimize the query, add an appropriate index, revise connection-pool limits through load testing, and introduce a dashboard showing pool saturation and query latency. A new canary analysis compares payment-service error rate, latency, and database utilization before broad deployment. An SLO-based release policy pauses promotion when the projected error-budget consumption exceeds the agreed threshold.

Outcome and Learning Points

The incident was not caused by Kubernetes resource exhaustion. The platform was functioning as configured, but an application change created dependency saturation. The case demonstrates why user-oriented SLIs, distributed traces, dependency metrics, recent-change correlation, and reversible mitigation are more effective than examining CPU alone. It also shows that a post-incident action should improve both the system and the organization’s ability to detect similar failures.

Post-Incident Learning and Reliability Improvement

A post-incident review should be blameless in the technical sense: it examines system conditions, decisions, incentives, and safeguards rather than treating individual error as the explanation. Blamelessness does not mean absence of accountability. Owners, deadlines, and verification criteria are still required for corrective actions.

The review should establish a factual timeline, quantify user impact and SLO consumption, identify contributing factors, evaluate detection and response, and record what worked as well as what failed. Root cause should not be reduced to the final technical defect. A database query may be the immediate cause, while insufficient load testing, weak canary analysis, missing pool saturation metrics, and an unsafe timeout policy are contributing conditions.

Actions should be prioritized by risk reduction. They may include code changes, improved tests, capacity adjustments, safer deployment controls, better alerts, runbook updates, dependency contracts, access changes, or training. “Monitor more closely” is not a sufficiently specific action. A useful action names the measurement, owner, deadline, and success condition.

Challenges, Risks and Governance

Observability introduces its own risks. Telemetry can contain personal data, credentials, business secrets, or identifiers that permit user tracking. Access should follow least privilege, transport and storage should be protected, and retention should reflect legal, contractual, and operational requirements. Redaction should occur as close to the source as practical, while ensuring that security teams retain the evidence needed for investigations.

Cost is another constraint. High-cardinality metrics, unbounded logs, full-fidelity traces, and long retention periods can make an observability platform expensive. Cost controls include aggregation, adaptive sampling, tiered storage, retention classes, metric reviews, and ownership tagging. Cost reduction must not eliminate the data needed for high-severity incidents.

Common operational mistakes include alerting on every available metric, using averages for tail latency, ignoring dependency health, failing to propagate trace context, treating restarts as a diagnosis, changing multiple components simultaneously, and retaining no incident evidence after recovery. Another mistake is measuring infrastructure availability instead of the actual customer journey.

The platform must also tolerate observability failure. Collectors can become overloaded, exporters can fail, and a monitoring backend can be unavailable during the incident it is meant to diagnose. Buffering, redundant collectors, health checks, independent synthetic monitoring, and documented fallback procedures improve resilience.

Emerging Directions

Current practice is moving toward unified telemetry pipelines, service catalogs, continuous profiling, eBPF-based network and system visibility, and automated correlation of metrics, logs, traces, and events. Machine-learning techniques can identify unusual patterns and assist query construction, but anomaly detection does not replace an SLO or an incident hypothesis. Automated remediation must be constrained by approval policies, blast-radius controls, audit trails, and rollback mechanisms.

OpenTelemetry is increasingly important as a common instrumentation and collection layer. In parallel, platform teams are treating observability as an internal product: they provide standard libraries, naming conventions, dashboards, alert policies, ownership metadata, and paved paths for application teams. This approach reduces duplicated effort while preserving service-specific expertise.

Best Practices

A mature Kubernetes observability practice should:

• Begin with user journeys and SLOs rather than tool installation.
• Instrument applications consistently and propagate correlation identifiers.
• Use metrics for trends and alerts, logs for detailed events, traces for request paths, and events for platform state changes.
• Control metric cardinality and telemetry retention deliberately.
• Page on urgent, actionable symptoms and review alert quality regularly.
• Maintain dashboards and runbooks for critical services.
• Use reversible mitigation and preserve evidence during incidents.
• Test observability through failure injection, load tests, and incident exercises.
• Review error budgets during release and capacity decisions.
• Convert post-incident findings into owned, verifiable engineering work.

Chapter Summary

Observability enables engineers to infer the behaviour of distributed containerized systems from metrics, logs, traces, events, and profiles. Prometheus provides time-series metrics and alert evaluation, Grafana supports analysis and visualization, and OpenTelemetry supplies vendor-neutral instrumentation and telemetry pipelines. Their value depends on coherent naming, contextual metadata, correlation, cardinality control, and appropriate retention.

SRE frames reliability through SLIs, SLOs, SLAs, and error budgets. User-impacting symptoms and rapid error-budget consumption should guide alerting and release decisions. Production troubleshooting requires establishing impact, identifying scope, correlating recent changes, testing hypotheses, mitigating safely, and verifying recovery. Post-incident reviews should be blameless, evidence-based, and focused on measurable improvements. Security, privacy, cost, governance, and observability-platform resilience must be treated as part of the design rather than afterthoughts.

Key Terms

Observability: The ability to understand internal system states from externally produced telemetry.

Monitoring: The collection and evaluation of known system conditions, often for alerting.

Metric: A numerical measurement recorded over time.

Log: A timestamped record describing an application, platform, or security event.

Trace: A representation of a request’s journey through distributed components.

Span: A timed operation within a distributed trace.

Event: A record of a state change, such as a Kubernetes scheduling failure.

Cardinality: The number of distinct values or time series created by metric labels.

PromQL: Prometheus Query Language for selecting and aggregating time-series data.

OpenTelemetry: A vendor-neutral framework for telemetry instrumentation and collection.

SLI: A quantitative indicator of a service’s observed behaviour.

SLO: A target value or range for an SLI over a specified period.

SLA: An externally communicated service commitment, often contractual.

Error budget: The permitted amount of unreliability implied by an SLO.

Golden signals: Traffic, errors, latency, and saturation used for service diagnosis.

Incident commander: The person coordinating priorities, decisions, and response during an incident.

Tail latency: High-percentile response time affecting the slowest requests.

Blameless post-incident review: A factual review focused on system conditions and learning rather than personal blame.

Review Questions

Short Answer Questions

1. How does observability differ from conventional monitoring?

2. What is the purpose of trace context propagation?

3. Define an SLI and give one suitable example for an API.

4. Why can unrestricted metric labels create operational problems?

5. What responsibilities does an incident commander perform?

Long Answer Questions

1. Explain how Prometheus, Grafana, and OpenTelemetry can work together in a Kubernetes observability architecture.

2. Discuss the relationship among SLIs, SLOs, SLAs, and error budgets.

3. Describe a systematic method for diagnosing a distributed application incident.

4. Explain the design principles for useful metrics, logs, and traces.

5. Discuss the security, privacy, cost, and governance challenges of production observability.

Analytical / Application Questions

1. An API has normal CPU utilization but rising p99 latency and timeout errors. Explain how you would use metrics, traces, logs, and Kubernetes events to investigate.

2. A team receives frequent alerts for CPU above 80 percent, although users experience no failures. Evaluate the alerting design and propose a better approach.

3. A new deployment causes intermittent failures only for one region and one version of a service. Construct an incident-response and mitigation plan using SLOs, change correlation, and controlled hypothesis testing.

22. Multiple Choice Questions with Answers

Question 1

Which telemetry type is most directly suited to showing the path and latency contribution of a request across several microservices?

A. Metric

B. Distributed trace

C. Kubernetes event

D. Node audit record

Correct Answer: B. Distributed trace

Question 2

What is the principal operational danger of placing unrestricted request IDs in Prometheus metric labels?

A. The metrics become impossible to timestamp

B. The labels prevent alert evaluation

C. They create excessive time-series cardinality

D. They automatically expose database passwords

Correct Answer: C. They create excessive time-series cardinality

Question 3

A service has a 99.9% availability SLO over a 30-day period. What does the remaining 0.1% represent?

A. The service-level agreement penalty

B. The permitted error budget for the period

C. The maximum CPU utilization

D. The required trace-sampling rate

Correct Answer: B. The permitted error budget for the period

Question 4

Which alert is most closely aligned with SRE principles?

A. Page whenever any node reaches 70% CPU

B. Page whenever a pod is recreated once

C. Page when rapid error-budget consumption indicates significant user impact

D. Page whenever a log message has warning severity

Correct Answer: C. Page when rapid error-budget consumption indicates significant user impact

Question 5

During an incident, traces show checkout requests waiting on a payment-service database connection pool. What is the most appropriate immediate response?

A. Increase all Kubernetes namespaces indiscriminately

B. Mitigate through a reversible rollback or feature disablement and verify recovery

C. Delete all historical logs to reduce storage load

D. Ignore the dependency because the checkout pod is running

Correct Answer: B. Mitigate through a reversible rollback or feature disablement and verify recovery

Question 6

Which statement best describes OpenTelemetry?

A. A Kubernetes scheduler replacement

B. A vendor-neutral framework for telemetry instrumentation and collection

C. A database engine for Prometheus labels

D. A mandatory Grafana dashboard format

Correct Answer: B. A vendor-neutral framework for telemetry instrumentation and collection

Question 7

Why are latency histograms often preferable to averages for production analysis?

A. They eliminate the need for logs

B. They reveal the distribution and tail of request latency

C. They always require less storage than counters

D. They measure only successful requests

Correct Answer: B. They reveal the distribution and tail of request latency

Question 8

What should normally be the first focus of an incident investigation?

A. Changing several configuration values simultaneously

B. Identifying user impact, scope, and time of onset

C. Restarting every pod in the cluster

D. Deleting the newest deployment immediately

Correct Answer: B. Identifying user impact, scope, and time of onset

Question 9

Which post-incident action is the most actionable?

A. Improve reliability sometime later

B. Monitor the system more carefully

C. Add a pool-saturation dashboard owned by the payment team, with a defined deadline and verification test

D. Ask engineers to be more attentive

Correct Answer: C. Add a pool-saturation dashboard owned by the payment team, with a defined deadline and verification test

Question 10

Why should telemetry access and retention be governed carefully?

A. Telemetry can contain sensitive personal, security, or business information

B. Governance prevents all metric aggregation

C. Retention makes traces mathematically inaccurate

D. Access controls are relevant only to source code

Correct Answer: A. Telemetry can contain sensitive personal, security, or business information