Metrics, Logs, Traces, Events, and Telemetry

Introduction

Cloud systems are dynamic, distributed, and economically variable. Resources can be created automatically, applications may span several regions, and a single user request can cross load balancers, containers, databases, queues, and external services. In such environments, administrators cannot manage reliability or cost by inspecting individual servers. They require telemetry: machine-generated evidence about the behaviour, health, performance, usage, and cost of a system.

Telemetry is commonly divided into metrics, logs, traces, and events. These forms of evidence are related but not interchangeable. Metrics support efficient measurement and alerting; logs preserve detailed records; traces reveal the path and timing of a request; and events record significant state changes. Effective cloud monitoring combines all four with appropriate instrumentation, collection intervals, retention policies, data-quality controls, and governance.

This chapter establishes the foundation for selecting and interpreting telemetry in infrastructure, applications, networks, databases, containers, and business systems. It also explains how telemetry supports cost optimization without encouraging harmful practices such as indiscriminate data collection or reducing observability below a safe level.

Learning Objectives

By the end of this chapter, you should be able to:

1. Differentiate metrics, logs, traces, events, and the broader concept of telemetry.
2. Select meaningful infrastructure, application, network, database, container, and business metrics.
3. Explain instrumentation, collection intervals, aggregation, sampling, and retention.
4. Assess telemetry quality using accuracy, completeness, timeliness, consistency, and uniqueness.
5. Relate observability data to reliability, troubleshooting, capacity planning, and cloud cost control.
6. Design a practical monitoring approach for a distributed cloud application.

1. Telemetry and Observability

Telemetry is the automated collection, transmission, storage, and analysis of measurements and records produced by a system. The word originally referred to measurement at a distance; in cloud computing it includes data emitted by software, operating systems, networks, platforms, and managed services.

Observability is the ability to infer the internal state of a system from its externally available outputs. Telemetry is the evidence; observability is the capability to use that evidence to answer questions such as:

• Is the service available?
• Which component is causing increased latency?
• Did a deployment introduce errors?
• Which resources are underutilized?
• Is a cost increase caused by demand, configuration, or abnormal activity?

Monitoring usually refers to the routine collection of known indicators and the generation of alerts. Observability is broader: it includes investigation of unexpected conditions and the ability to explore relationships across components.

A useful operational model is:

Signal → Context → Interpretation → Action

For example, a high checkout error rate is a signal. Trace identifiers and deployment information provide context. Correlation with database connection exhaustion supports interpretation. Rollback, scaling, or configuration correction is the action.

2. The Four Principal Telemetry Types

2.1 Metrics

A metric is a numerical observation associated with a name, time, unit, and usually a set of dimensions or labels. Examples include CPU utilization, request rate, memory bytes, queue depth, and database transaction latency.

Metrics are efficient because many observations can be aggregated into compact time series. They are therefore suitable for dashboards, alerting, capacity planning, service-level objectives, and cost analysis.

Common metric forms include:

• Gauge: a value that may rise or fall, such as current memory usage.
• Counter: a cumulative value that increases, such as total requests or bytes transmitted. Counters may reset when a process restarts.
• Rate: change in a counter over time. If a counter changes from C₁ to C₂ during an interval Δt, the average rate is:

Here, r is the rate, C₁ and C₂ are counter values, and Δt is elapsed time. If C is measured in requests and Δt in seconds, r is requests per second. The formula assumes the counter has not reset; reset handling is required in real systems.

• Histogram: a distribution represented through buckets or another summarized structure, useful for latency and request-size analysis.
• Summary: a client-side statistical summary, often including count and selected quantiles. Summaries from separate instances cannot always be combined accurately.

Averages alone can be misleading. If nine requests take 10 ms and one takes 1,000 ms, the average is 109 ms, but most users experience approximately 10 ms while one user experiences severe delay. Percentiles are often more informative. The p95 latency is the value below which approximately 95% of observations fall, subject to the definition and approximation method used by the monitoring system.

2.2 Logs

A log is a time-stamped record of an occurrence or state, usually containing a message and contextual fields. Logs may describe authentication attempts, application errors, configuration changes, or database operations.

Structured logs use fields such as JSON rather than embedding all information in free text. A useful application log may include:

• timestamp in UTC;
• severity;
• service and version;
• environment;
• request or trace identifier;
• operation name;
• outcome and error code;
• latency or relevant numerical values;
• carefully controlled user or transaction identifiers.

Logs are highly detailed but can be expensive to store, index, and query. Excessive debug logging can increase storage, network, and ingestion costs while making important records harder to find. Logging sensitive data can also create privacy and security risks.

2.3 Traces

A trace represents the journey of one logical operation through a distributed system. It is composed of one or more spans. A span describes an operation, its start and end time, status, attributes, and parent-child relationship.

For example, an online purchase trace might contain spans for the API gateway, authentication service, inventory service, payment provider, and database. Trace context is propagated between services, commonly through request headers. A trace identifier connects spans belonging to the same request, while a span identifier identifies one operation.

Traces are particularly valuable for:

• locating latency in a multi-service request;
• identifying dependencies and service bottlenecks;
• understanding retries and fan-out;
• connecting user-visible failures to infrastructure causes.

Tracing every request may be costly. Sampling can be applied at the head of a request, before its outcome is known, or at the tail, after information about errors and latency is available. Tail sampling can retain unusual or failed traces more effectively but requires more processing and buffering.

2.4 Events

An event records a meaningful occurrence or state transition. Examples include a virtual machine being stopped, a deployment completing, a scaling policy adding instances, a certificate approaching expiry, or a database failover beginning.

Events differ from ordinary logs because they usually represent notable changes that deserve correlation or action. Events are often sparse and operationally significant. An event stream may trigger automation, such as opening an incident, applying a policy, or starting a remediation workflow.

2.5 Comparison

These types complement one another. A metric may reveal elevated latency, a trace may locate the slow dependency, a log may explain the error, and an event may show that a deployment occurred immediately beforehand.

3. Selecting Meaningful Metrics

A metric is useful only when it supports a decision. Selecting every available metric creates noise, cost, and alert fatigue. A practical selection process is:

1. Identify the service or business objective.
2. Identify user-visible outcomes and failure modes.
3. Select indicators that explain those outcomes.
4. Define dimensions needed for diagnosis.
5. Establish thresholds, baselines, or service-level objectives.
6. Confirm that collection cost and privacy implications are acceptable.

3.1 Infrastructure Metrics

Infrastructure metrics describe compute, storage, and platform resources. Important examples include:

• CPU utilization and CPU throttling;
• memory utilization, allocation, and swap activity;
• disk capacity, read/write operations, throughput, and latency;
• network bytes, packets, errors, drops, and connection counts;
• host availability and restart frequency;
• virtual machine or node count;
• resource requests, limits, and actual usage.

Utilization is not automatically a performance problem. High CPU utilization may be efficient if latency and error objectives remain satisfied. Conversely, low CPU utilization does not prove that a system is healthy if it is blocked on storage or constrained by memory.

3.2 Application Metrics

Application metrics should represent behaviour meaningful to users and operators. Examples include:

• request rate;
• success and error rates by operation;
• latency distributions rather than only averages;
• active sessions or jobs;
• queue age and processing delay;
• cache hit ratio;
• retry rate and timeout rate;
• dependency availability;
• completed transactions and rejected transactions.

A common reliability model is the four golden signals: latency, traffic, errors, and saturation. These provide a compact starting point for service monitoring.

3.3 Network Metrics

Network monitoring should examine both volume and quality. Useful measurements include throughput, round-trip latency, packet loss, retransmissions, connection failures, DNS resolution time, load-balancer response codes, and network-interface errors. For a distributed application, dimensions such as source region, destination region, availability zone, protocol, and service pair may be valuable.

Network metrics must be interpreted carefully. A rise in traffic may indicate business growth, a retry storm, an attack, or a misconfigured client. A rise in latency may be caused by distance, congestion, TLS negotiation, DNS, or an overloaded downstream service.

3.4 Database Metrics

Relevant database measurements include:

• query latency and query error rate;
• transactions per second;
• active connections and connection-pool utilization;
• lock waits and deadlocks;
• buffer or cache hit ratio;
• replication lag;
• storage growth and I/O latency;
• CPU and memory pressure;
• slow-query counts.

Database metrics should be connected to application operations. A database may show acceptable average latency while a particular query used by checkout becomes slow. Query identifiers, carefully selected query fingerprints, and traces can provide this connection without recording sensitive values.

3.5 Container and Orchestration Metrics

Container environments require measurements at several levels:

• container restarts and termination reasons;
• CPU and memory usage versus requests and limits;
• throttling and out-of-memory kills;
• pod readiness and scheduling failures;
• node pressure and eviction counts;
• replica availability;
• deployment rollout progress;
• image-pull failures.

A container that is repeatedly restarted may produce a low average CPU value while still causing an outage. Availability and lifecycle metrics are therefore as important as resource utilization.

3.6 Business Metrics

Business metrics connect technology to organizational outcomes. Examples include orders per minute, payment success rate, customer sign-ups, revenue per completed transaction, abandoned carts, and cost per transaction.

Business metrics should be defined precisely. “Orders” might mean orders submitted, paid orders, or fulfilled orders. A definition should specify the event, time window, inclusion rules, and source of truth. Business metrics often require stronger access controls than ordinary technical metrics.

4. Instrumentation and Collection Architecture

Instrumentation is the addition of measurement and recording capability to software or infrastructure. It may be:

• Automatic, supplied by an operating system, cloud platform, agent, or service mesh;
• Library-based, using a standard client library for metrics, logs, or traces;
• Manual, where developers explicitly record a business operation or important state;
• Sidecar or collector-based, where telemetry is collected outside the application process.

A typical architecture contains four stages:

1. Production: applications, hosts, containers, databases, and cloud services emit telemetry.
2. Collection: agents or collectors receive, enrich, filter, batch, and forward data.
3. Storage and processing: time-series databases, log stores, trace backends, and event buses retain and index data.
4. Use: dashboards, alerts, investigations, reports, and automated actions consume the data.

Open standards and vendor-neutral formats can reduce dependence on one monitoring provider. However, interoperability does not eliminate the need to understand each backend’s limits, pricing, sampling behaviour, and query semantics.

Instrumentation should preserve context. Consistent service names, environment labels, version identifiers, timestamps, and correlation identifiers make cross-signal analysis possible. Instrumentation should also avoid high-cardinality labels such as unrestricted user IDs or request IDs in metric dimensions. High cardinality can create millions of time series and sharply increase cost and query difficulty. Such identifiers are usually more appropriate in logs or traces.

5. Collection Intervals, Aggregation, and Sampling

The collection interval is the time between measurements or scrapes. A shorter interval can detect brief failures and provide more detailed data, but it increases processing, storage, and network costs. A longer interval reduces cost but may hide short-lived saturation.

The appropriate interval depends on the phenomenon and decision. For example:

• infrastructure capacity planning may use one-minute or five-minute data;
• rapid failure detection may require seconds;
• billing analysis may use hourly or daily aggregation;
• audit records should generally be event-driven rather than periodically sampled.

Aggregation reduces volume but can destroy important information. Averaging latency can hide outliers. Summing request counts across instances is usually meaningful, while averaging instance-level request rates may not be, especially when traffic is unevenly distributed.

Sampling selects a subset of observations. It is useful for traces and verbose logs, but sampling policies should preserve rare failures and high-latency operations. A practical policy may retain all error traces, a high proportion of slow traces, and a lower percentage of successful routine traces.

When using aggregated data, operators must know whether values are cumulative, interval-based, or calculated by the backend. Counter resets, missing intervals, clock differences, and late-arriving records can affect results.

6. Retention and Data Lifecycle

Retention is the period for which telemetry remains available. A sound policy classifies data according to operational value, legal requirements, security risk, and cost.

A common pattern is:

• short retention for detailed debug logs and full traces;
• moderate retention for operational metrics and incident evidence;
• longer retention for capacity trends and approved audit records;
• archival or deletion when the business and compliance purpose ends.

Retention should not be chosen solely by storage price. Longer retention increases the impact of unauthorized access and may create privacy obligations. Logs containing personal data may require minimization, masking, access controls, deletion procedures, and regional storage restrictions.

Compression, tiered storage, downsampling, and selective indexing reduce cost. However, compressed or downsampled data may not support the same investigations as the original records. Retention policies should therefore be tested against realistic incident scenarios.

7. Telemetry Data Quality

Poor-quality telemetry can produce confident but incorrect decisions. Important quality dimensions include:

• Accuracy: does the value represent the real condition?
• Completeness: are expected records present?
• Timeliness: does the data arrive quickly enough for its purpose?
• Consistency: are names, units, timestamps, and meanings stable?
• Uniqueness: are records duplicated?
• Validity: does the value conform to an expected range and type?
• Lineage: can the origin and transformation of the data be identified?

Time synchronization is essential. Hosts and services should use reliable clock sources and consistent time zones, normally UTC. Distributed traces also require careful handling of clock skew; parent-child relationships and collector processing times can help identify uncertainty.

Units must be explicit. Bytes and bits, milliseconds and seconds, decimal and binary prefixes, and percentage points versus percentages are frequent sources of error. A metric named memory_usage is weaker than one defined as container_memory_working_set_bytes with documented semantics.

Telemetry pipelines should themselves be monitored. Useful pipeline metrics include collector queue depth, dropped records, export failures, ingestion latency, backend throttling, and schema-validation failures. If monitoring data is missing during an incident, the monitoring system has become an operational dependency and must be designed accordingly.

8. Using Telemetry for Cost Optimization

Observability supports cost optimization in several ways:

• identifying idle or underutilized resources;
• comparing resource requests with actual consumption;
• detecting unexpected data transfer;
• measuring storage and log-ingestion growth;
• relating cloud spend to transactions or customers;
• evaluating the effect of autoscaling and rightsizing.

A simple unit-cost metric is:

The formula is meaningful only when the numerator and denominator use the same time period and clearly defined allocation rules. Shared costs may require allocation by usage, ownership, or an agreed business rule. The result is not a universal accounting truth; it is a management indicator whose interpretation depends on the allocation model.

Cost reduction must not be pursued by blindly deleting telemetry. Removing logs may reduce ingestion cost but increase incident duration and operational risk. Better approaches include filtering redundant records, reducing unnecessary label cardinality, sampling routine traces, using appropriate retention tiers, and monitoring the cost of observability itself.

9. Illustrative Case: Diagnosing an Expensive and Slow Checkout Service

Background

An online retailer operates a containerized checkout service across two availability zones. During a promotional campaign, cloud expenditure rises by 38 percent and customers report slow checkout. The platform team initially sees moderate average CPU utilization and concludes that the system is adequately provisioned.

Problem

The organization must determine whether the problem is caused by insufficient compute, database contention, network behaviour, inefficient retries, or an observability configuration that has itself become expensive.

Analysis

The team examines four signals. The application metric for checkout p95 latency has increased from 420 ms to 2.1 s, while average latency has increased only modestly. Error metrics show a rise in payment timeouts. Traces reveal that the payment-provider span is slow and that clients retry the entire checkout operation. Database metrics show increased connection usage caused by those retries. Network metrics show increased outbound requests to the provider.

Logs confirm that a recent client-library change retries timeouts three times. An event record identifies the deployment time, which matches the beginning of the incident. Cost data shows that outbound transfer, payment calls, and log ingestion increased. The logging system is recording full request payloads at debug level, although those payloads are not needed for diagnosis.

Approach and Solution

The team rolls back the client-library change, limits retries with exponential backoff and a maximum attempt count, and introduces an idempotency key so that repeated payment requests do not create duplicate charges. It retains traces for errors and slow requests but samples routine successful traces. Debug payload logging is disabled, and sensitive fields are removed from retained records.

Dashboards are revised to show checkout success rate, p95 and p99 latency, payment-provider latency, retry rate, connection-pool utilization, outbound request rate, and cost per completed checkout. Alerts are based on user-visible objectives rather than CPU alone.

Outcome

Checkout latency and timeout rates return near their previous levels. Database connection pressure falls, outbound traffic decreases, and log-ingestion cost declines. The organization avoids adding capacity that would have treated the symptom rather than the cause.

Learning Points

1. Average utilization cannot establish service health by itself.
2. Percentiles reveal user-impacting tail latency.
3. Traces connect application symptoms to downstream dependencies.
4. Logs and deployment events provide causal context.
5. Retry behaviour can multiply both technical load and cloud cost.
6. Cost optimization is safest when linked to reliability and business metrics.

10. Best Practices and Limitations

Effective telemetry design follows several principles:

• Begin with user, service, and business objectives.
• Use consistent names, units, timestamps, and service identities.
• Prefer structured logs and meaningful dimensions.
• Keep metric cardinality controlled.
• Instrument critical paths and dependencies.
• Record errors and unusual latency with higher sampling priority.
• Monitor the telemetry pipeline itself.
• Test alert quality and investigate false positives.
• Document retention, ownership, access, and deletion rules.
• Protect secrets and personal data through minimization and redaction.
• Review observability cost as part of cloud-financial management.

Telemetry has limitations. It may be incomplete, delayed, sampled, distorted by clock differences, or affected by instrumentation overhead. Correlation does not automatically prove causation. A deployment event occurring near an outage is evidence for investigation, not proof that the deployment caused it. Human interpretation, controlled testing, and knowledge of system design remain necessary.

Artificial intelligence can assist with anomaly detection, log summarization, and incident correlation, but it should not be treated as an unquestionable source of truth. Models may learn normal behaviour from periods that already contain failures, misclassify rare events, or expose sensitive data. Human review, explainable evidence, access control, and auditability remain essential.

Chapter Summary

Telemetry is the evidence used to understand and operate cloud systems. Metrics provide efficient numerical time series for trends and alerts; logs preserve detailed records; traces show the path and timing of distributed requests; and events identify significant state changes. These signals are complementary rather than interchangeable.

Meaningful metric selection begins with service and business objectives. Infrastructure, application, network, database, container, and business metrics should describe outcomes, capacity, dependencies, saturation, and cost. Instrumentation must provide consistent context while controlling cardinality and protecting sensitive information.

Collection intervals, aggregation, sampling, and retention involve trade-offs among resolution, cost, performance, and investigative value. Telemetry quality depends on accuracy, completeness, timeliness, consistency, validity, uniqueness, and lineage. Monitoring the telemetry pipeline is essential because missing or delayed telemetry can compromise operations.

Observability can support cost optimization by identifying waste, measuring unit cost, and exposing inefficient retries or data transfer. However, cost reduction must preserve sufficient evidence for reliability, security, compliance, and incident response.

Key Terms

Telemetry: Machine-generated measurements and records collected from a system.

Observability: The ability to infer internal system state from external outputs.

Metric: A numerical, time-associated observation.

Gauge: A metric representing a value that can increase or decrease.

Counter: A cumulative metric that normally increases until reset.

Histogram: A distribution representation using buckets or summarized ranges.

Log: A time-stamped record of an occurrence or state.

Structured log: A log whose fields follow a machine-readable structure.

Trace: The end-to-end record of one logical request.

Span: A timed operation within a trace.

Event: A record of a significant occurrence or state transition.

Instrumentation: Code or tooling that produces telemetry.

Cardinality: The number of distinct values or time series created by dimensions.

Sampling: Selecting a subset of telemetry observations.

Retention: The period for which telemetry is stored and accessible.

Percentile: A distribution value below which a specified proportion of observations fall.

Saturation: The extent to which a resource or service is approaching a limiting capacity.

Data lineage: The documented origin and transformation history of data.

Service-level objective: A target level of reliability or performance for a service.

Unit cost: Cost allocated to one defined business or technical unit.

Review Questions

Short Answer Questions

1. What is the principal difference between a metric and a log?
2. What is the purpose of a span within a distributed trace?
3. Why can high-cardinality metric labels increase cost?
4. Name four dimensions of telemetry data quality.
5. Why should the telemetry pipeline be monitored?

Long Answer Questions

1. Compare metrics, logs, traces, and events, explaining the operational questions each can answer.
2. Explain how collection interval, aggregation, sampling, and retention affect observability cost and investigative value.
3. Discuss appropriate metrics for a containerized application that uses a relational database.
4. Explain how telemetry can support cloud cost optimization without compromising reliability.
5. Describe an instrumentation and collection architecture for a distributed cloud application.

Analytical / Application Questions

1. A service has normal average latency but rapidly increasing p99 latency and timeout errors. Which telemetry should be examined first, and why?
2. A cloud bill increases after a deployment, while CPU utilization remains stable. Develop a telemetry-based investigation plan.
3. Design a retention and sampling policy for a payment service, considering reliability, privacy, compliance, and cost.

Multiple Choice Questions with Answers

Question 1

Which telemetry type is most directly suited to showing the path and timing of one request across multiple services?

A. Metric

B. Log

C. Trace

D. Event

Correct Answer: C. Trace

Question 2

Which metric form represents a value that may increase or decrease, such as current memory usage?

A. Gauge

B. Counter

C. Trace

D. Event

Correct Answer: A. Gauge

Question 3

What is a major risk of using unrestricted user IDs as metric labels?

A. Lower clock accuracy

B. High metric cardinality

C. Loss of all logs

D. Automatic data encryption

Correct Answer: B. High metric cardinality

Question 4

Which signal is generally most useful for identifying that a deployment occurred immediately before an outage?

A. Deployment event

B. CPU gauge alone

C. Disk capacity forecast

D. Business definition

Correct Answer: A. Deployment event

Question 5

If a counter changes from 1,200 requests to 1,800 requests over 60 seconds, what is the average rate?

A. 10 requests per second

B. 20 requests per second

C. 30 requests per second

D. 60 requests per second

Correct Answer: B. 20 requests per second

Question 6

Why are latency percentiles often preferred to averages?

A. They eliminate the need for timestamps

B. They show aspects of the latency distribution and tail behaviour

C. They always require less storage

D. They prove the cause of an outage

Correct Answer: B. They show aspects of the latency distribution and tail behaviour

Question 7

Which database metric most directly indicates waiting caused by competing transactions?

A. Lock waits

B. Image-pull failures

C. DNS record count

D. Container restart time

Correct Answer: A. Lock waits

Question 8

Which practice is most appropriate for reducing routine trace cost while preserving important incidents?

A. Delete all traces

B. Sample successful traces and retain errors or slow traces preferentially

C. Collect only CPU metrics

D. Store every trace forever

Correct Answer: B. Sample successful traces and retain errors or slow traces preferentially

Question 9

A cost-per-transaction measure is meaningful only when which condition is satisfied?

A. The numerator and denominator refer to clearly defined and matching periods

B. CPU utilization is exactly 100 percent

C. All logs are retained indefinitely

D. Every request has identical latency

Correct Answer: A. The numerator and denominator refer to clearly defined and matching periods

Question 10

Which quality dimension asks whether expected telemetry records are present?

A. Completeness

B. Uniqueness

C. Latency percentile

D. Cardinality

Correct Answer: A. Completeness