12. Monitoring, Logging, Performance and Troubleshooting

Introduction and Context

Linux administration is not complete when a service has been installed and started. A production administrator must determine whether the system is healthy, whether it is meeting its service objectives, how its behaviour is changing, and why it fails when it does. Monitoring, logging, performance analysis and troubleshooting therefore form a connected discipline rather than a collection of commands.

Monitoring observes system state and trends. Logging preserves an event record. Performance analysis explains how resources are being consumed. Troubleshooting applies structured reasoning to identify and correct faults. Together, these activities provide operational visibility and support evidence-based decisions about reliability, capacity and risk.

In DevOps environments, this discipline extends beyond individual servers. Applications are distributed across virtual machines, containers, networks, databases and cloud services. A slow response may be caused by CPU saturation, memory pressure, storage latency, packet loss, a failed dependency or an inefficient application query. Effective administrators therefore correlate metrics, logs, traces, configuration and deployment history rather than relying on a single symptom.

Learning Objectives

After studying this chapter, students should be able to:

• distinguish metrics, logs, traces and audit evidence;
• select meaningful indicators for Linux hosts and services;
• use Linux tools to investigate CPU, memory, storage, network and process behaviour;
• interpret systemd journals and persistent log files;
• apply a disciplined troubleshooting and root-cause-analysis process;
• distinguish symptoms, contributing factors and underlying causes;
• perform basic capacity planning and evidence-based tuning;
• design monitoring and logging practices that respect security, privacy and governance requirements.

Observability as an Evidence System

A metric is a numerical measurement associated with time, such as CPU utilisation, load average, available memory, disk latency or HTTP error rate. Metrics are efficient for dashboards, alerting and trend analysis. They are, however, summaries: a metric may indicate that a failure occurred without explaining the exact sequence of events.

A log is a timestamped record of an event or message. Examples include an SSH login, a failed systemd unit, a kernel warning or an application exception. Logs provide detail but can be difficult to aggregate and interpret at scale. Their value depends on accurate timestamps, useful context, consistent severity and controlled retention.

A trace follows a request across components. In a distributed application, a trace can show that a request spent 20 milliseconds in a web service, 700 milliseconds waiting for a database and 300 milliseconds calling an external API. Trace context, usually represented by trace and span identifiers, permits events from separate services to be correlated.

Audit evidence answers questions of accountability: who performed an action, what was changed, when it occurred and whether it was authorised. Audit records should be protected against unauthorised alteration. They are not interchangeable with ordinary diagnostic logs, although a single event may have both operational and audit significance.

These forms of evidence complement one another. A rising request latency metric may lead an engineer to a trace; the trace may identify a database call; database and kernel logs may explain a storage error. This progression is often described as observability: the ability to infer internal system behaviour from externally collected evidence.

Important Measurements

CPU utilisation should be interpreted by mode. User time represents application work, system time represents kernel work, I/O wait indicates time during which runnable work is delayed by I/O, and steal time indicates time taken by a virtualisation host. High utilisation is not automatically a fault. A CPU-bound batch process may use 100 per cent of available CPU while meeting its objective, whereas moderate utilisation accompanied by rising latency may indicate a bottleneck elsewhere.

Load average represents the average number of runnable tasks and, on Linux, tasks in uninterruptible sleep, commonly associated with I/O. It is not a percentage and must be considered in relation to the number of logical CPUs. A load of 8 may be acceptable on a 32-CPU host but problematic on a 2-CPU host. Load average also reflects historical intervals, typically one, five and fifteen minutes, so it should not be treated as an instantaneous measure.

Memory analysis requires more than checking free memory. Linux uses available memory for file cache and reclaims it when applications require space. Important indicators include available memory, swap-in and swap-out activity, page faults, reclaim pressure and the resident memory of processes. Sustained swapping usually indicates memory pressure, although the presence of swap alone is not evidence of failure.

Storage performance is described through throughput, I/O operations per second, latency, queue depth, filesystem capacity and inode availability. A filesystem can have free bytes but no free inodes, preventing the creation of files. A full log partition can also stop applications from writing essential state. Disk utilisation reported by a tool is not identical to capacity utilisation: a device may have free space but be saturated by slow operations.

Network analysis considers link state, bandwidth, packet errors, drops, retransmissions, connection states, DNS resolution and service-level latency. A successful ping does not prove that an application protocol is healthy. Conversely, a blocked ICMP request does not necessarily indicate that HTTPS or SSH is unavailable.

A Practical Linux Evidence Toolkit

The following tools are useful because they expose different layers of the system:

• `uptime`, `top` and `htop` provide a rapid process and load overview.
• `ps aux` or `ps -eo pid,ppid,stat,%cpu,%mem,cmd` supports repeatable process inspection.
• `vmstat 1` displays runnable processes, memory, paging, system activity and CPU states over time.
• `free -h` reports memory and swap summaries; the `available` field is generally more useful than `free` alone.
• `iostat -xz 1` helps interpret device throughput, utilisation and latency when the sysstat package is available.
• `df -h` and `df -i` examine filesystem bytes and inodes, while `du` locates directories consuming space.
• `sar` records and retrieves historical CPU, memory, disk and network statistics when configured.
• `ss -tulpn` displays listening sockets and `ss -s` summarises socket states.
• `ip addr`, `ip route` and `ip -s link` examine addresses, routes and interface errors.
• `dig` tests DNS resolution, and `curl -v` tests an application endpoint while exposing protocol details.
• `journalctl` queries systemd’s journal; `systemctl status` and `systemctl show` expose unit state and properties.
• `strace` can reveal system calls, delays and failing files for a selected process, but should be used carefully in production.

Point-in-time commands can mislead. For example, `top` may capture a brief CPU spike, while a one-second interval from `vmstat` or `iostat` reveals whether the behaviour is sustained. Repeated measurements, baselines and timestamps make evidence more reliable.

Logging with systemd and the Linux Kernel

Modern Linux distributions commonly use systemd-journald. Useful queries include:

```bash

journalctl -b

journalctl -u nginx --since "30 minutes ago"

journalctl -p warning..alert

journalctl -k

journalctl -f

```

The first command displays messages from the current boot, the second filters by service and time, the third filters by priority, and the fourth focuses on kernel messages. `journalctl -f` follows new events and is useful during a controlled reproduction.

A production system should normally retain logs across reboot, restrict access to sensitive records and control journal growth. Rotation and retention policies should reflect operational need, storage capacity and legal requirements. Traditional services may write to `/var/log`, including authentication, kernel, boot and application records. Log rotation prevents unbounded growth and commonly compresses older files.

Log quality is an engineering concern. Messages should include a timestamp, severity, component, operation, request or correlation identifier and an actionable description. Passwords, private keys, session tokens and unnecessary personal information must not be recorded. Time synchronisation through a trusted source is essential when events from multiple machines are correlated.

Centralised logging protects evidence from loss when a host fails and enables cross-system searches. However, forwarding logs introduces network, storage and access-control requirements. Critical audit records should be transmitted securely, assigned appropriate retention, and protected from alteration. Administrators must distinguish a log pipeline failure from an absence of application events: silence may mean either “nothing happened” or “collection stopped.”

A Reasoning Framework for Troubleshooting

Troubleshooting should be treated as a controlled investigation rather than trial-and-error configuration. A useful sequence is:

1. Define the impact. Identify affected users, services, locations, time period and business consequence.

2. Stabilise the situation. Protect data, reduce further harm and apply a reversible mitigation if necessary.

3. Collect evidence. Record commands, timestamps, configuration versions, recent changes and relevant metrics and logs.

4. Form hypotheses. Link symptoms to plausible causes and identify evidence that would support or disprove each one.

5. Test one variable at a time. Prefer safe, reversible experiments and avoid changing multiple settings simultaneously.

6. Restore and verify. Confirm service health from the user’s perspective, not only from the server process state.

7. Identify root causes and prevent recurrence. Document contributing conditions, corrective actions and monitoring improvements.

This method avoids two common errors. The first is treating the most visible symptom as the cause—for example, restarting a service whenever requests fail. The second is making several changes at once, which may restore service but destroys the ability to determine which change mattered.

A fault tree can organise possible causes. If a web service is unavailable, branches may include process failure, listener failure, firewall or route failure, dependency failure, certificate expiry, resource exhaustion and application-level rejection. A timeline then relates these possibilities to deployments, configuration changes, alerts and external events.

Root-cause analysis should distinguish the immediate cause from systemic contributors. A process may have crashed because it exhausted memory; memory may have been exhausted because a deployment introduced a leak; the leak may have remained undetected because no alert measured working-set growth. Corrective action should address all relevant levels, including code, configuration, capacity and observability.

Diagnosing Common Resource Failures

CPU and Load Problems

Begin with `uptime` and `top`, then identify whether the load is caused by runnable CPU work or blocked I/O. Use `ps` to rank processes and `pidstat` where available to examine per-process CPU, context switching and I/O. High system CPU may suggest excessive system calls, networking, interrupt processing or kernel activity. High steal time in a virtual machine may require investigation by the hosting team rather than local tuning.

Before increasing CPU capacity, establish whether the process is inefficient, incorrectly parallelised, constrained by a quota or waiting on another resource. Profiling, query analysis and workload scheduling may provide better results than adding processors.

Memory Pressure

Use `free -h`, `vmstat`, process resident-set information and cgroup metrics where containers are involved. Look for sustained paging, an exhausted swap area, an OOM-killer message in the kernel log, and processes whose memory grows unexpectedly. The Linux OOM killer selects a process according to several factors; its action is a consequence of memory exhaustion, not proof that the selected process caused the leak.

Possible responses include correcting a leak, reducing concurrency, adjusting cache size, adding memory, or revising container limits. Disabling swap without analysis can make an incident worse. Memory tuning should be based on workload behaviour and service objectives.

Disk and Filesystem Failures

When a service cannot write, check both `df -h` and `df -i`. Use `du` to locate large directories, but remember that deleted files held open by a process may consume space without appearing in directory totals; `lsof +L1` can help identify them. Examine `dmesg` or `journalctl -k` for device errors and `iostat` for latency and queueing.

Do not delete files indiscriminately from a production filesystem. Confirm ownership, retention requirements and whether a process still needs the file. A full root filesystem, corrupted filesystem, failing device and slow network-mounted filesystem require different responses.

Network and Service Failures

Confirm the service state with `systemctl status`, then check listening sockets using `ss`. Test locally and remotely to separate application, host firewall, routing and upstream problems. Inspect `ip route`, DNS resolution, interface counters and connection states. For HTTP, use `curl` and inspect status codes, redirects, certificates and response time.

A service can be active while being functionally unavailable: it may listen on the wrong address, reject authentication, fail to reach a database or return errors for every request. Health checks must therefore test meaningful dependencies without creating excessive load.

Capacity Planning and Evidence-Based Tuning

Capacity planning connects observed demand with future requirements. A simple model records workload volume, resource consumption, peak behaviour, growth rate and a safety margin. If a service handles 1,000 requests per minute at 40 per cent CPU and demand is expected to double, linear extrapolation is only a first approximation. Caching, concurrency, contention and nonlinear saturation can change the result.

Useful planning indicators include peak and percentile latency, error rate, CPU saturation, memory working set, storage growth, network throughput and queue depth. Percentiles are important because averages can conceal a poor user experience: a service with acceptable mean latency may have severe tail latency for a small but significant group of requests.

Tuning should follow a hierarchy. First remove defects and unnecessary work. Next optimise algorithms, queries, indexes, concurrency and caching. Then adjust operating-system or service parameters. Finally add capacity or redesign the architecture. Every change should have a hypothesis, a measurable success criterion, a rollback plan and a comparison with baseline measurements.

Service-level objectives make performance decisions meaningful. An objective might specify availability, latency or error rate over a defined period. An alert should indicate a condition requiring action, not merely an interesting fluctuation. Alert fatigue develops when thresholds are too sensitive, duplicated or disconnected from user impact.

Applied Case: Intermittent Slowness in a Linux API Service

Background

This illustrative case concerns an API running under systemd on two Linux virtual machines. Users report that requests are usually fast but occasionally take several seconds. The service process remains active, and a basic health check reports success.

Situation and Analysis

The operations team first defines the impact: elevated tail latency occurs during business hours and affects a subset of requests. A dashboard shows that average CPU is 45 per cent, but the 95th-percentile latency has increased. This prevents the team from incorrectly concluding that the system is healthy from average CPU alone.

Traces reveal that slow requests spend most of their time waiting for a database call. Host metrics show bursts of disk latency on the database virtual machine. `iostat -xz 1` indicates a growing device queue during the same intervals, while application logs show repeated temporary database errors and retries. The timeline reveals that a reporting job was deployed shortly before the problem began.

Approach and Solution

The team temporarily schedules the reporting job outside peak API hours and reduces its concurrency, restoring latency. It then analyses the job query and finds that an index is missing. After testing the index in a controlled environment, the query is optimised and retry limits are adjusted so that transient failures do not multiply load. Monitoring is improved to include database latency, retry counts, queue depth and API tail latency.

Outcome

The immediate symptom—slow API responses—is removed without indiscriminately adding CPU to the API machines. The deeper cause was an inefficient reporting query, while excessive concurrency and retries were contributing factors. The case demonstrates why application traces, database logs and host metrics must be correlated.

Learning Points

The service being “up” did not mean that it was meeting its objective. Average CPU did not expose the bottleneck. A reversible mitigation protected users while evidence was collected, and the permanent solution addressed workload design, indexing, concurrency and observability.

Risks, Governance and Professional Practice

Monitoring can create privacy and security risks. Logs may contain identities, addresses, commands, request data or confidential business information. Collection should follow data minimisation, purpose limitation, access control and retention requirements. Sensitive fields should be redacted before centralisation, and access should be audited.

Operational access must be least-privileged. Diagnostic tools can expose environment variables, command arguments, network content or other users’ data. Capturing packets or tracing processes may require authorisation and should be limited in duration and scope. Audit records should be protected from modification, while administrators should preserve relevant evidence during security incidents rather than overwriting it through uncontrolled remediation.

Automation is valuable for anomaly detection, log enrichment, remediation and capacity forecasting. Automated actions must have bounded permissions, clear failure modes and human review for high-impact changes. Machine-learning alerts can generate false positives or miss novel failures; they supplement, rather than replace, engineering judgement and reliable telemetry.

Current Developments and Best Practices

Cloud-native systems increasingly use OpenTelemetry-style instrumentation to connect metrics, logs and traces through common context. eBPF-based tools can observe kernel and network behaviour with limited application changes, although their use requires attention to overhead, permissions and data governance. Containers and orchestration platforms add dimensions such as pod restarts, resource limits, node pressure and control-plane health.

A mature operational practice generally follows these principles:

• establish baselines before declaring abnormal behaviour;
• monitor user-visible service objectives as well as host resources;
• collect consistent timestamps and correlation identifiers;
• centralise important logs and protect them against loss and alteration;
• alert on actionable, service-relevant conditions;
• retain enough historical data for trend and capacity analysis;
• make changes incrementally, measure results and preserve rollback paths;
• practise incident reviews without blame, focusing on system improvement;
• test monitoring, backups and recovery rather than assuming they work.

Chapter Summary

Monitoring, logging, tracing and audit evidence provide complementary views of Linux systems. Metrics reveal trends and thresholds, logs describe events, traces follow requests across components, and audit records establish accountability. Effective diagnosis combines these sources with repeatable measurements from tools such as `top`, `vmstat`, `iostat`, `df`, `ss`, `journalctl` and `systemctl`.

Troubleshooting is an evidence-driven process: define impact, stabilise the service, collect facts, test hypotheses, verify recovery and address root causes. CPU, memory, disk and network symptoms must be interpreted in context, because high utilisation is not automatically failure and low utilisation does not guarantee good service. Capacity planning uses workload trends, percentiles, saturation indicators and service objectives. Secure logging, privacy protection, least privilege, controlled automation and continuous learning are essential to professional operations.

Key Terms

Metric: A time-associated numerical measurement of system or service behaviour.

Log: A timestamped record describing an event, message or condition.

Trace: A record of a request’s path through one or more system components.

Audit evidence: Protected information showing who performed an action, what occurred and when.

Observability: The ability to infer internal system state from external telemetry.

Load average: A measure of runnable and uninterruptible tasks averaged over time.

I/O wait: CPU time associated with waiting for input/output completion.

Memory pressure: A condition in which available memory is insufficient for workload demand.

Swap: Disk-backed space used to extend virtual memory.

Inode: A filesystem data structure representing a file and its metadata.

Queue depth: The number of operations waiting for a resource.

Tail latency: High-percentile response time, such as the 95th or 99th percentile.

Baseline: A reference description of normal system behaviour.

Correlation identifier: A value used to connect related events across components.

Root cause: The underlying condition whose removal prevents recurrence of a failure.

Capacity planning: Estimating future resource requirements from demand and growth evidence.

Service-level objective: A measurable target for availability, latency or another service property.

Alert fatigue: Reduced attention caused by excessive, low-value or inaccurate alerts.

Review Questions

Short Answer Questions

1. Why is the `available` memory value generally more useful than `free` memory alone?

2. What information does `iostat` provide during storage troubleshooting?

3. Distinguish a log from an audit record.

4. Why should load average be interpreted alongside CPU count?

5. What is the purpose of a correlation identifier?

Long Answer Questions

1. Explain how metrics, logs and traces complement one another during diagnosis of a distributed service.

2. Describe a structured troubleshooting process from incident definition through root-cause analysis.

3. Explain how Linux administrators can investigate a filesystem that appears unable to accept new files.

4. Discuss the relationship between capacity planning, percentiles, saturation and service-level objectives.

5. Analyse the security, privacy and governance requirements of centralised logging.

Analytical / Application Questions

1. A server shows a load average of 12, four logical CPUs, low user CPU and high I/O wait. What evidence would you collect next, and what hypotheses would you test?

2. An API process is active and listening, but remote users receive intermittent failures. Develop a diagnostic sequence using service, network, log and dependency evidence.

3. A container is repeatedly killed while host memory appears plentiful. Explain how cgroups, process metrics, kernel logs and resource limits could be used to determine the cause.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a metric from a log?

A. A metric is numerical and time-associated, while a log records an event or message

B. A metric always contains more detail than a log

C. A log can only be produced by the kernel

D. A metric is used exclusively for security auditing

Correct Answer: A. A metric is numerical and time-associated, while a log records an event or message

Question 2

A Linux host has a load average of 10 on a machine with two logical CPUs. Which interpretation is most appropriate?

A. The host is certainly using exactly 10 percent CPU

B. The host may have substantial runnable or uninterruptible work relative to its CPU capacity

C. The host has exactly 10 failed processes

D. The host has 10 gigabytes of memory pressure

Correct Answer: B. The host may have substantial runnable or uninterruptible work relative to its CPU capacity

Question 3

Which command is most directly useful for checking filesystem inode exhaustion?

A. df -i

B. free -h

C. ss -s

D. uptime

Correct Answer: A. df -i

Question 4

An application is active according to systemctl but users cannot connect. What should the administrator check next?

A. Only the system hostname

B. Listening sockets, bind address, firewall and remote connectivity

C. The number of installed text editors

D. The shell history of every user

Correct Answer: B. Listening sockets, bind address, firewall and remote connectivity

Question 5

Why can average latency be misleading when evaluating a service?

A. Averages cannot be calculated from measurements

B. Average latency may conceal poor tail latency affecting a subset of requests

C. Average latency measures only CPU utilisation

D. Average latency is always higher than the 99th percentile

Correct Answer: B. Average latency may conceal poor tail latency affecting a subset of requests

Question 6

Which sequence best represents disciplined troubleshooting?

A. Change several settings, reboot, and document the result

B. Define impact, collect evidence, test hypotheses, verify recovery, and address causes

C. Delete logs, restart all services, and wait for improvement

D. Increase capacity before determining the bottleneck

Correct Answer: B. Define impact, collect evidence, test hypotheses, verify recovery, and address causes

Question 7

A filesystem has free bytes but applications cannot create files. Which condition is a plausible explanation?

A. All inodes are exhausted

B. The CPU has entered user mode

C. The network interface has a valid address

D. The system has too much available memory

Correct Answer: A. All inodes are exhausted

Question 8

In the illustrative API case, what evidence identified the principal bottleneck?

A. High API host CPU on every request

B. Traces and storage metrics showing time spent in a database call and elevated database disk latency

C. A failed DNS lookup on the client workstation

D. An empty system journal

Correct Answer: B. Traces and storage metrics showing time spent in a database call and elevated database disk latency

Question 9

Which practice best protects sensitive information in centralised logs?

A. Record passwords to simplify incident response

B. Apply data minimisation, redaction, access control and suitable retention

C. Allow every administrator unrestricted access

D. Disable timestamps to reduce privacy risk

Correct Answer: B. Apply data minimisation, redaction, access control and suitable retention

Question 10

What is the main purpose of a baseline in performance analysis?

A. To provide a reference for distinguishing normal and abnormal behaviour

B. To guarantee that a service will never fail

C. To replace all application traces

D. To measure only filesystem ownership

Correct Answer: A. To provide a reference for distinguishing normal and abnormal behaviour