Cloud Telemetry and Observability Architecture

Introduction

Modern cloud systems are distributed across virtual machines, containers, serverless functions, managed databases, queues, APIs, networks and third-party services. Their behaviour cannot be understood reliably from a single server console or a small set of infrastructure graphs. A request may cross several services, regions and trust boundaries before producing a result. Consequently, cloud engineers require a structured system for collecting, transporting, storing, correlating and interpreting operational evidence.

This evidence is called telemetry. Observability is the broader engineering capability of using telemetry to infer the internal state of a system from its externally available outputs. A mature observability architecture does not merely collect large quantities of data. It produces information that is relevant, attributable, timely, secure and economically sustainable.

This chapter develops an architecture for metrics, logs, traces, events and profiles. It explains instrumentation, semantic conventions, tagging, sampling, retention, correlation and OpenTelemetry. It also considers operational cost, governance, privacy, reliability and the practical limitations of observability platforms.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Distinguish metrics, logs, traces, events and profiles and select each for an appropriate investigative purpose.
2. Design a cloud-native telemetry pipeline from signal production to analysis and action.
3. Apply instrumentation and consistent resource, service and business attributes.
4. Explain trace context propagation, correlation and the role of OpenTelemetry.
5. Design sampling and retention policies that balance diagnostic value, compliance and cost.
6. Evaluate observability architecture using reliability, security, privacy and economic criteria.
7. Apply these principles to a realistic distributed-cloud scenario.

1. Telemetry and Observability as Engineering Capabilities

Telemetry is the automated production and transmission of measurements or records from a system. The principal telemetry signals are:

Observability is not identical to monitoring. Monitoring commonly evaluates known conditions against thresholds, objectives or alerts. Observability includes monitoring but also supports investigation of previously unknown failure modes. A system with many dashboards may still have poor observability if its signals cannot be correlated to a customer request, deployment, tenant or business operation.

Three practical properties are especially important:

• High-cardinality usability: telemetry can be associated with a particular service, instance, customer-safe tenant identifier, region or request without making storage and query costs unmanageable.
• Contextual correlation: signals share identifiers and attributes so that an engineer can move from an alert to a trace, from a trace to logs, and from a deployment to its effects.
• Actionability: collected data supports a decision, such as rollback, capacity adjustment, remediation or escalation.

Observability is therefore an architectural property, not a product feature alone. A vendor platform can store and display data, but application design, instrumentation quality and operational practice determine whether the data is useful.

2. The Five Core Signals

2.1 Metrics

A metric is a numerical observation recorded over time. Common metric forms include counters, gauges, histograms and summaries.

• A counter increases monotonically, except when reset, and is suitable for requests processed or bytes transmitted.
• A gauge represents a value that may rise or fall, such as queue depth or memory utilization.
• A histogram records observations in buckets and is useful for latency or payload size distributions.
• A summary may calculate quantiles at the producer, although quantiles from separate instances cannot generally be aggregated exactly without additional distribution information.

Metrics are efficient for alerting and trend analysis. However, aggregating data can hide individual failures. A service-wide error-rate metric may reveal a problem but not identify the affected request or dependency.

For a measured interval, a basic request error ratio is:

The numerator and denominator must refer to the same time interval, service boundary and request population. If 240 of 12,000 requests fail, the ratio is 240 ÷ 12,000 = 0.02, or 2%. This ratio is meaningful only if retries and synthetic traffic are treated consistently.

Latency should usually be represented as a distribution rather than only an average. The arithmetic mean is:

where xᵢ is an observation and n is the number of observations. The mean can be distorted by a small number of very slow requests; therefore, percentile estimates, histograms and maximum values may be needed for user-facing latency analysis.

2.2 Logs

A log is a timestamped record describing an occurrence. Structured logs use fields such as timestamp, severity, service name, operation, message, trace identifier and error type rather than embedding all information in free text.

A useful log record is:

{
  "timestamp": "2025-02-15T10:20:31.402Z",
  "severity": "ERROR",
  "service.name": "payment-api",
  "event.name": "charge_declined",
  "trace_id": "...",
  "deployment.environment": "production",
  "error.type": "CardDeclined"
}

Logs are valuable for detailed diagnosis, audit evidence and security investigation. They can become expensive when applications emit verbose messages, stack traces repeatedly or unbounded user attributes. Sensitive data such as passwords, access tokens, payment-card numbers and unnecessary personal information must not be logged.

2.3 Traces

A distributed trace represents the path of one operation through a system. A trace contains one or more spans. A span has a start time, duration, operation name, status, attributes and relationships to other spans. Parent-child relationships commonly represent synchronous calls; links can represent other causal relationships, such as asynchronous message processing.

Trace duration for a sequential path is approximately the elapsed time from the first root-span start to its completion. It must not be calculated by simply adding all span durations when spans overlap in parallel. A trace can expose queue delay, network delay, database time, retries and fan-out behaviour that a service-level metric cannot show.

Trace identifiers should be propagated across service boundaries using a standardized context mechanism. Instrumentation must preserve context through HTTP calls, messaging, asynchronous tasks and thread or process boundaries where feasible.

2.4 Events

An event records a meaningful state transition or domain occurrence, such as OrderAccepted, DeploymentCompleted or CertificateExpiring. Events differ from ordinary logs because they often have defined schemas, consumers and business significance. An event can be operational, security-related or domain-oriented.

Events should include an event name, schema version, occurrence time, producer, subject or resource, and correlation identifiers. They should not be confused with alerts: an event states that something happened, whereas an alert represents an evaluated condition requiring attention.

2.5 Profiles

Profiling samples program execution to estimate where CPU time, memory allocation, lock contention or other resources are consumed. Continuous profiling can reveal performance regressions that are difficult to diagnose from traces alone. Profilers must be configured carefully because excessive sampling overhead, sensitive stack information or unsuitable retention can create operational and privacy risks.

3. A Cloud-Native Telemetry Pipeline

A telemetry architecture can be understood as a sequence of stages:

1. Instrumentation: an application, runtime, platform or agent creates signals.
2. Collection: a local agent, SDK, collector or provider integration receives signals.
3. Processing: data is enriched, filtered, transformed, redacted, sampled or batched.
4. Transport: data is sent through a reliable protocol or message path.
5. Storage: signals are stored in systems optimized for time series, search, trace analysis or profiling.
6. Analysis: dashboards, queries, anomaly detection and service maps support interpretation.
7. Action: alerts, tickets, automation, rollback or human response follow.

Figure: A cloud telemetry pipeline from signal production to operational action.

A collector can run beside each workload, as a node-level agent, or as a gateway service. Local collection reduces application coupling and can provide buffering, while gateways centralize policy and export management. A robust design considers backpressure, queue limits, retry behaviour, dead-letter handling and failure isolation. Telemetry must not be allowed to consume unlimited application CPU, memory or network bandwidth.

4. Instrumentation and OpenTelemetry

Instrumentation may be:

• Zero-code or automatic, supplied by an agent, runtime integration or platform service.
• Manual or code-based, where developers create spans, metrics or events around business operations.
• Infrastructure-based, where a cloud provider or host agent reports resource and platform measurements.

Automatic instrumentation is efficient for common libraries but may miss business meaning. Manual instrumentation can record an operation such as checkout.validate_inventory, but it must avoid excessive span creation and unstable naming. Names should describe operation types rather than contain unbounded identifiers; an identifier belongs in an attribute when permitted.

OpenTelemetry is a vendor-neutral observability framework and ecosystem for generating, collecting and exporting telemetry. It provides APIs, SDKs, semantic conventions, propagators and the OpenTelemetry Collector. It is not itself a storage backend or a complete incident-management product.

The Collector commonly uses a pipeline containing receivers, processors and exporters. For example, an OTLP receiver may accept data, a memory limiter may protect the collector, a batch processor may reduce network overhead, and exporters may deliver data to a metrics, log or trace backend. Exact component availability and supported features depend on the implementation and version, so production designs should verify current documentation.

Instrumentation should be tested like application code. Tests should check that:

• service and environment attributes are present;
• trace context survives calls and asynchronous work;
• errors have appropriate status and type;
• metrics have stable names and units;
• sensitive fields are excluded or redacted;
• telemetry volume remains within an agreed budget.

5. Naming, Attributes, Tags and Cardinality

A telemetry record usually has a small set of resource attributes identifying the producing entity, such as service name, service version, deployment environment, cloud region and host or container identity. Instrumentation attributes describe an operation, such as HTTP method, route template, database system or messaging destination.

A tag or attribute adds dimensions to a signal. Tags enable filtering and grouping, but every additional dimension can increase storage, indexing and query cost. High-cardinality values include raw user IDs, session IDs, request IDs and unbounded URLs. They are often appropriate for traces or selected logs, but dangerous as metric labels.

Good practice includes:

• use route templates such as /customers/{customerId} rather than raw paths as metric dimensions;
• use controlled vocabularies for environment, region, status and service role;
• keep tenant identifiers pseudonymous and apply access controls;
• put request identifiers in traces and logs, not generally in metric labels;
• document units, temporality and aggregation for each metric;
• version schemas when fields or meanings change.

Cardinality is not merely a storage issue. It affects dashboard usability, alert stability and the ability to compare data across services. A metric with millions of nearly unique time series is usually a sign that identity data has been placed in the wrong signal.

6. Correlation Across Signals

Correlation allows an engineer to navigate from a symptom to evidence. A common path is:

1. A latency or error metric triggers an alert.
2. The engineer filters traces for the affected service and time interval.
3. A trace identifies a slow database span or failed downstream call.
4. The trace identifier retrieves related structured logs.
5. Deployment or configuration events explain when the behaviour changed.

Correlation requires consistent clocks, timestamps, resource attributes and identifiers. Trace context should be propagated through supported protocols. Logs emitted during a span should include the active trace and span identifiers. Events should include a correlation or causation identifier where appropriate.

Correlation is not causation. Two signals sharing a time interval or deployment version does not prove that one caused the other. Engineers should combine telemetry with controlled experiments, dependency knowledge, change records and, where possible, reproducible tests.

7. Sampling, Retention and Cost Control

Telemetry design is also a cost-management problem. A simple storage-volume estimate is:

where V is uncompressed volume, r is records per second, s is average bytes per record, and t is the number of seconds retained. The equation assumes a stable rate and average size and excludes compression, indexing, replication, metadata and provider-specific billing rules.

For example, 2,000 log records per second with an average encoded size of 1,000 bytes retained for 24 hours produce:

2,000 × 1,000 × 86,400 = 172,800,000,000 bytes,

approximately 172.8 GB in decimal units before indexing, replication and compression. Actual charges may be based on ingestion, indexed volume, query scans, network transfer or retention tiers, so the calculation is an estimate rather than an invoice forecast.

Sampling keeps a subset of telemetry. Head sampling decides early, often when a trace begins. Tail sampling decides after observing the trace, allowing retention of errors, long traces or unusual patterns. Tail sampling usually requires buffering and more collector resources.

A policy might retain:

• 100% of failed, slow or security-relevant traces;
• a lower probabilistic percentage of ordinary successful traces;
• 100% of service-level metrics needed for alerting;
• structured logs at information level for a short period and detailed logs only during controlled diagnosis.

Sampling must not remove the measurements needed to calculate service objectives. If a compliance investigation requires complete audit records, those records must not be treated like ordinary sampled diagnostic traces.

Retention should be tiered. Recent high-value data may remain in fast searchable storage; older data may move to lower-cost storage; data with no operational, legal or analytical purpose should be deleted. Retention schedules must specify signal, purpose, storage class, duration, owner and deletion method.

8. Reliability, Security and Governance

Telemetry pipelines are production systems. They require capacity planning, health metrics, dashboards and alerts for dropped records, queue saturation, export failures, collector restarts and backend throttling. At-least-once delivery may create duplicates; exactly-once delivery is difficult and often unavailable end-to-end. Consumers should therefore tolerate duplicates where possible.

Security controls include encryption in transit and at rest, least-privilege export credentials, network segmentation, tenant isolation and role-based query access. Logs and traces may contain personal or confidential information through URLs, exception messages, SQL statements or payload attributes. Data classification should occur before collection, not only after ingestion.

Important governance questions include:

• What is the legitimate purpose for collecting this field?
• Is the field necessary, or can it be generalized or pseudonymized?
• Who may query it, and are queries audited?
• How long must it be retained?
• In which jurisdiction is it stored?
• Can it be deleted when a valid request or policy requires deletion?

Observability can also create surveillance risks when employee actions, customer identifiers or detailed behavioural traces are retained without proportionate purpose. Ethical design requires necessity, transparency, access control and minimization.

9. Applied Case: Illustrative Online Retail Platform

Background

An online retailer operates a web front end, product catalogue, inventory service, payment service and order queue across two cloud regions. During peak campaigns, customers report intermittent checkout failures. Existing monitoring shows CPU utilization and a general error count, but engineers cannot identify the failing dependency quickly.

Problem Analysis

The system has unstructured application logs, inconsistent service names and no propagated trace context. The checkout service records raw URLs containing customer identifiers. Metrics use request IDs as labels, generating excessive time series. Payment errors are mixed with validation errors, and no event identifies deployment completion.

The immediate operational problem is slow diagnosis. The financial problem is unnecessary telemetry ingestion and indexing. The governance problem is exposure of personal information.

Approach and Solution

The team first defines service ownership, naming conventions and a telemetry schema. Each workload receives stable resource attributes for service, version, environment and region. HTTP routes use templates. Request IDs are retained only in traces and selected logs; they are removed from metric labels.

OpenTelemetry libraries instrument inbound HTTP requests, outbound calls, database operations and queue producers and consumers. Context propagation links the browser-facing request to the payment and order-processing spans. Structured logs include severity, event name, trace identifier and error classification, but payment-card data and authentication tokens are explicitly excluded.

A collector runs near workloads for local buffering and redaction, while a gateway collector performs tail sampling. All error traces and traces exceeding the agreed latency threshold are retained. A controlled sample of successful traces is retained for baseline analysis. Metrics for request rate, error ratio, queue age and latency distributions are retained at a resolution suitable for service-level objectives.

Deployment events are emitted with version, service, region, actor or automation identity, and change reference. Dashboards link error-ratio alerts to traces and deployment events. Access to customer-related telemetry is restricted and audited. Short-lived searchable storage is used for detailed logs, with longer retention only for approved audit events.

Outcome

In a controlled load test, the team can identify that a payment dependency becomes slow immediately after a configuration change. The trace shows queue delay and downstream duration separately, while correlated logs identify the returned error class. The corrected metric dimensions reduce the number of time series, and redaction removes sensitive values before export. The architecture improves diagnostic speed without retaining every raw record indefinitely.

Learning Points

1. A dashboard is insufficient when signals cannot be correlated.
2. Business-level manual instrumentation complements automatic instrumentation.
3. Sampling should preserve rare, slow and failed operations.
4. Metric cardinality must be designed deliberately.
5. Privacy and cost controls belong in the collection pipeline, not as an afterthought.

10. Contemporary Developments and Professional Practice

Cloud providers increasingly offer managed OpenTelemetry ingestion, service maps, anomaly detection, continuous profiling and integrated application-performance monitoring. These services can reduce operational effort but may increase dependence on proprietary schemas, query languages or pricing models. A portability strategy should distinguish open instrumentation from backend-specific features.

AI-assisted operations can summarize incidents, cluster similar logs, suggest likely causes and generate queries. Such systems should be treated as decision support rather than authoritative diagnosis. Engineers must validate the evidence, control access to sensitive telemetry, monitor hallucinated explanations and preserve human accountability for remediation.

A practical observability maturity path is:

1. establish ownership, naming and basic health metrics;
2. instrument critical request paths and propagate context;
3. centralize structured logs and correlate them with traces;
4. introduce meaningful events, profiling and service objectives;
5. apply evidence-based sampling, tiered retention and cost budgets;
6. automate safe remediation while continuously testing data quality and governance.

Chapter Summary

Cloud telemetry consists of metrics, logs, traces, events and profiles, each answering different operational questions. Observability is the ability to infer system state from these signals and to investigate both known and unknown failure modes. A telemetry pipeline moves data from instrumentation through collection, processing, transport, storage, analysis and action.

OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions, propagation mechanisms and collectors, but it does not replace a backend or operational practice. Effective instrumentation uses stable names, useful resource attributes, controlled cardinality and consistent context propagation. Correlation links alerts, traces, logs, events and deployments into an investigative path.

Sampling, retention and tiered storage are essential for economic sustainability, but must preserve service-objective measurements, audit evidence and rare failures. Telemetry pipelines require reliability engineering, security, privacy protection, access control and data minimization. Finally, observability should be evaluated by the decisions it enables, not by the volume of data collected.

Key Terms

Telemetry: Automated production and transmission of measurements or records from a system.

Observability: The capability to infer internal system state from externally available outputs.

Metric: A numerical observation recorded over time.

Counter: A monotonically increasing metric, subject to reset.

Gauge: A metric representing a value that may increase or decrease.

Histogram: A distribution-oriented metric that records observations in buckets.

Log: A timestamped record describing an occurrence.

Trace: A representation of one distributed operation across services.

Span: A timed unit of work within a trace.

Event: A record of a meaningful state transition or occurrence.

Profile: Statistical runtime evidence about resource or code consumption.

Instrumentation: Code, agent or integration that produces telemetry.

Semantic convention: A defined naming and attribute convention for interoperable telemetry.

Cardinality: The number of distinct values or time series created by dimensions.

Sampling: Selecting a subset of telemetry for collection or retention.

Retention: The period and storage policy for keeping telemetry.

Correlation: Linking signals through identifiers, timestamps and shared attributes.

OpenTelemetry: An open-source, vendor-neutral observability framework and ecosystem.

Tail sampling: Sampling performed after enough of a trace has been observed to evaluate it.

Resource attribute: An attribute identifying the entity that produced telemetry.

Review Questions

Short Answer Questions

1. What is the difference between monitoring and observability?
2. Distinguish a counter, gauge and histogram.
3. Why are raw request IDs usually unsuitable as metric labels?
4. What information does a distributed trace provide that a single service metric may not?
5. Name two security controls required for telemetry pipelines.

Long Answer Questions

1. Explain the five major telemetry signals and identify a suitable use for each.
2. Describe the stages of a cloud-native telemetry pipeline and the responsibilities of a collector.
3. Explain how OpenTelemetry supports portability and why it does not constitute a complete observability platform by itself.
4. Discuss how sampling and retention policies can reduce cost without damaging incident investigation.
5. Analyse the relationship between structured logging, trace context propagation and cross-signal correlation.

Analytical / Application Questions

1. A service emits 500 log records per second, each averaging 2,000 bytes, for 12 hours. Estimate the uncompressed decimal storage volume and state two reasons the actual billed volume may differ.
2. Design a telemetry policy for a payment service handling personal data. Specify attributes to collect, fields to exclude, sampling rules, access controls and retention categories.
3. A dashboard shows a high checkout error ratio, but traces are unavailable for successful requests and logs lack trace identifiers. Diagnose the architectural weaknesses and propose a phased improvement plan.

Multiple Choice Questions with Answers

Question 1

Which telemetry signal is most directly suited to showing where time was spent across several distributed services?

A. A gauge metric

B. A distributed trace

C. A deployment event

D. A runtime profile

Correct Answer: B. A distributed trace

Question 2

What is the principal risk of using raw customer IDs as metric labels?

A. They prevent metrics from having timestamps

B. They make counters decrease

C. They create excessive cardinality and cost

D. They automatically encrypt the metric

Correct Answer: C. They create excessive cardinality and cost

Question 3

Which OpenTelemetry component commonly receives, processes and exports telemetry?

A. The OpenTelemetry Collector

B. A DNS resolver

C. A database index only

D. A load balancer listener

Correct Answer: A. The OpenTelemetry Collector

Question 4

A system emits 1,000 records per second, each averaging 500 bytes, for 3,600 seconds. What is the estimated uncompressed volume?

A. 1.8 MB

B. 18 MB

C. 180 MB

D. 1.8 GB

Correct Answer: C. 180 MB

Question 5

Which sampling approach can retain traces selectively after evaluating whether they contain errors or unusually high latency?

A. Head sampling

B. Tail sampling

C. Round-robin routing

D. Metric aggregation

Correct Answer: B. Tail sampling

Question 6

Which statement best describes an event?

A. A numerical value that is always monotonic

B. A record of a meaningful state transition

C. A statistical sample of CPU stacks only

D. A storage encryption key

Correct Answer: B. A record of a meaningful state transition

Question 7

Why should route templates generally be used instead of raw URLs in metric dimensions?

A. Templates reduce unbounded cardinality

B. Templates remove all latency information

C. Raw URLs cannot be logged

D. Templates disable trace propagation

Correct Answer: A. Templates reduce unbounded cardinality

Question 8

Which practice best protects sensitive information in telemetry?

A. Log all request payloads for completeness

B. Place access tokens in span attributes

C. Redact or exclude unnecessary sensitive fields before export

D. Use a larger dashboard

Correct Answer: C. Redact or exclude unnecessary sensitive fields before export

Question 9

Why is adding all span durations incorrect for a trace containing parallel operations?

A. Spans never have durations

B. Parallel spans may overlap in time

C. Trace identifiers cannot be stored

D. Metrics must replace traces

Correct Answer: B. Parallel spans may overlap in time

Question 10

What is the main purpose of propagating trace context between services?

A. To correlate work belonging to one distributed operation

B. To increase metric cardinality deliberately

C. To delete logs after one minute

D. To convert gauges into counters

Correct Answer: A. To correlate work belonging to one distributed operation