Operations, Observability, and Troubleshooting

Introduction

A container is not operationally successful merely because it was built successfully. A useful container must start reliably, remain healthy, communicate with its dependencies, store or retrieve data correctly, consume acceptable resources, and generate evidence that enables operators to understand its behaviour. Operations therefore connects container construction with dependable service delivery.

This chapter presents a systematic approach to observing and troubleshooting Docker-based workloads. It covers logs, events, metrics, health checks, resource consumption, storage, networking, startup failures, and performance problems. The emphasis is on evidence-based diagnosis rather than trial and error.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish logs, events, metrics, traces, and health status.
2. Inspect container state using Docker commands.
3. Design useful container health checks.
4. Diagnose build, startup, storage, networking, and performance failures.
5. Apply a structured troubleshooting workflow.
6. Interpret resource limits and container statistics.
7. Explain operational, security, governance, and ethical considerations.

1. The Operational Model of a Container

A container is a process, or group of processes, isolated by the host operating system. Its image supplies the filesystem and startup definition, while the runtime supplies namespaces, control groups, networking, mounts, and lifecycle management. Operational behaviour is therefore produced by several interacting layers:

A productive diagnosis moves between these layers without assuming that every failure is an application-code defect.

2. Observability: Evidence About Internal State

Monitoring usually asks whether known conditions remain within acceptable limits. Observability asks whether internal system state can be inferred from the outputs it produces. In container operations, the principal signals are logs, metrics, traces, events, and health status.

2.1 Logs

Logs are timestamped records of noteworthy events or application messages. Docker commonly captures a container's standard output and standard error through a logging driver. Basic commands include:

docker logs my-container
docker logs --follow --tail 100 my-container
docker logs --since 10m my-container
docker logs --timestamps my-container

A good operational log states what happened, when it happened, and enough contextual information to identify the request, component, or operation involved. Structured formats such as JSON make filtering and aggregation easier. Logs should distinguish severity levels such as DEBUG, INFO, WARN, and ERROR.

Logs are not a complete observability solution. They may be absent when a process writes only to an unexpected file, may be misleading when timestamps are inconsistent, and may become expensive when verbose output is uncontrolled. Never place passwords, access tokens, private keys, or unnecessary personal information in logs.

2.2 Events

Events describe changes in Docker object state, such as creation, starting, stopping, killing, health-status changes, or network attachment. They are particularly useful for correlating a failure with an operational action.

docker events

docker events --filter container=my-container

A log might show that an application failed to bind a port; an event may show that the container was restarted repeatedly after that failure. Events are usually short-lived operational records, whereas logs are application or process output.

2.3 Metrics

Metrics are numerical measurements collected over time. Common container metrics include CPU usage, memory usage, network bytes, block I/O, restart count, request rate, error rate, and latency.

docker stats

docker stats --no-stream my-container

A useful metric must have a name, unit, timestamp, and usually labels such as service, environment, and instance. Metrics support trend analysis and alerting, but an average can hide severe outliers. For latency, percentiles such as p95 or p99 are often more informative than the mean.

For a request service, a basic error rate can be expressed as:

The result is dimensionless; multiplying by 100 gives a percentage. This measure is meaningful only when the numerator and denominator refer to the same time interval and request population.

A simple CPU utilisation estimate over an interval is:

Container tools may report CPU usage relative to one CPU or as a percentage across available CPUs, depending on the tool and platform. Always verify the reporting convention before interpreting a value.

2.4 Health Checks

A health check is an automated test that reports whether a service is functioning sufficiently for its intended use. It is not identical to process liveness. A process can remain alive while being unable to serve requests, reach a dependency, or respond within an acceptable time.

A Dockerfile example is:

HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
  CMD wget --quiet --tries=1 --spider http://127.0.0.1:8080/health || exit 1

The endpoint should be inexpensive and deterministic. A liveness check generally asks whether the process should be restarted. A readiness-style check asks whether the service should receive traffic. Combining every dependency into a liveness check can cause restart storms: a temporary database outage may repeatedly restart an otherwise recoverable application.

Health checks have states such as starting, healthy, and unhealthy. They should have realistic startup allowances and timeouts. A check that is too strict creates false failures; one that is too weak creates false confidence.

2.5 Traces and Correlation

A distributed request may cross several containers. Logs from each component become far more useful when they carry a common request or trace identifier. Tracing systems represent a request as a trace containing spans for operations such as API handling, database access, and message publication. Docker alone does not provide complete distributed tracing, but containers can run instrumented applications and telemetry agents.

3. A Systematic Troubleshooting Method

Unstructured troubleshooting often changes several variables at once and destroys evidence. A disciplined method is more reliable.

Step 1: Define the Symptom

Describe the observable problem precisely: “The API returns HTTP 503 after deployment” is more useful than “the container is broken.” Record time, affected service, frequency, recent changes, and impact.

Step 2: Establish Scope

Determine whether the failure affects one container, one host, one network, one deployment, or all environments. Compare a healthy instance with a failing instance.

Step 3: Preserve Evidence

Capture logs, events, container inspection output, image identifiers, configuration versions, and relevant resource measurements before restarting or deleting objects. A restart may remove valuable state.

Step 4: Form and Rank Hypotheses

Possible causes should be consistent with the evidence. Rank hypotheses by likelihood, impact, and ease of testing. Test one significant change at a time.

Step 5: Test Narrowly

Use the smallest diagnostic action that distinguishes between hypotheses. For example, test DNS resolution separately from application authentication, and test local connectivity separately from external connectivity.

Step 6: Correct and Verify

Apply the smallest safe correction, verify health and functionality, and observe the service long enough to detect recurrence.

Step 7: Document and Prevent Recurrence

Record cause, evidence, corrective action, and preventive improvement. A post-incident improvement might be a stronger health check, a configuration validation step, or an alert on disk exhaustion.

A useful principle is: observe before modifying, isolate before escalating, and verify after changing.

4. Inspecting Container State

The following commands form a practical inspection toolkit:

docker ps
docker ps -a
docker inspect my-container
docker top my-container
docker port my-container
docker image inspect my-image
docker network inspect my-network
docker volume inspect my-volume

docker ps -a reveals containers that exited, while docker inspect exposes configuration and runtime state in machine-readable JSON. Important fields include the image identifier, command, environment, mounts, network settings, exit code, restart count, and health status.

An exit code is evidence, not a complete explanation. Exit code 0 normally indicates successful termination, but a long-running service that exits with code 0 may still be incorrectly configured. Non-zero codes indicate failure according to the application or shell convention. Exit code 137 often corresponds to termination by signal 9 and may be associated with an out-of-memory kill, but it should be confirmed using host and runtime evidence rather than assumed.

5. Diagnosing Common Failure Classes

5.1 Build Failures

Build failures occur before runtime. Common causes include an incorrect build context, missing files excluded by .dockerignore, invalid package references, network or registry failures, architecture incompatibility, and incorrect build arguments.

Useful practices include:

• Read the first meaningful error, not only the final failure message.
• Build with readable progress when detailed output is required.
• Confirm the build context contains the expected source files.
• Pin important dependency versions where reproducibility matters.
• Use multi-stage builds to separate compilation tools from the runtime image.
• Scan images for known vulnerabilities and excessive packages.

A successful build does not prove that the image will start. Build validation should be followed by a basic runtime test.

5.2 Startup and Configuration Failures

A container that exits immediately may have an invalid command, missing executable permission, incorrect working directory, absent environment variable, incompatible architecture, or an application that terminates after completing a one-time task.

Inspect the configured command and entrypoint. Run the image interactively when safe:

docker run --rm -it --entrypoint /bin/sh my-image

This is a diagnostic technique, not a substitute for a correct production command. Check environment values without exposing secrets, confirm file permissions, and verify that the main process runs in the foreground. A backgrounded application can make the container appear to exit even though a child process was launched.

Graceful shutdown also matters. The main process should receive termination signals and stop within the configured grace period. Poor signal handling can cause dropped requests and slow deployments.

5.3 Storage Failures

Container writable layers are ephemeral by design. Data that must survive container replacement belongs in a volume or an external data service. Diagnose storage using:

docker system df
docker inspect my-container
docker volume ls
docker volume inspect data-volume
docker exec my-container df -h

Typical failures include mounting a directory over files supplied by the image, incorrect ownership, read-only filesystems, full volumes, and writing data to a path that was not mounted. The host may also run out of inodes even when apparent byte capacity remains.

Do not delete unused volumes or images during an incident without confirming their ownership and recovery implications. Cleanup commands can destroy data.

5.4 Networking Failures

Separate networking questions into layers:

1. Is the process listening on the expected address and port?
2. Is the container attached to the correct network?
3. Can the client resolve the service name?
4. Can the client establish a connection?
5. Does the protocol exchange succeed?
6. Does authentication or authorization succeed?

Commands such as the following help isolate layers:

docker port my-container
docker network inspect app-net
docker exec client-container getent hosts api
docker exec client-container wget -qO- http://api:8080/health

EXPOSE documents an intended port; it does not publish the port to the host. Port publishing is configured with -p host_port:container_port. Applications that listen only on 127.0.0.1 inside a container may be unreachable through the container network; services commonly need to listen on the appropriate container interface.

5.5 Resource and Performance Failures

Use docker stats to inspect CPU, memory, network, and block I/O. High CPU may indicate an inefficient algorithm, a tight retry loop, excessive garbage collection, or insufficient CPU allocation. High memory may indicate a leak, an unexpectedly large workload, caching, or an unsuitable runtime limit. High I/O may arise from logging, database operations, or temporary-file use.

Resource limits protect hosts and improve fairness, but they do not automatically improve application performance. A memory limit that is too low can cause termination; a CPU limit that is too restrictive can increase latency. Establish a baseline under representative load before selecting limits.

For a service with a target response-time objective, latency should be observed alongside throughput and error rate. A system may show low average latency while a small but important group of requests experiences severe delay. Resource measurements should therefore be correlated with application-level metrics.

6. Worked Diagnostic Example

Suppose an API container repeatedly becomes unhealthy after deployment. Operators observe HTTP 503 responses from the reverse proxy.

First, they inspect state:

docker ps
docker inspect api --format '{{json .State.Health}}'

The health output shows repeated failures connecting to 127.0.0.1:8080. Application logs show that the API is listening on port 8080, but network inspection reveals that the application was configured to bind to a different interface and the health command was using an invalid path.

The team tests the process directly inside the container and then tests the endpoint from the proxy container. The process is alive, but /health returns 404 because the actual endpoint is /ready. The immediate correction is to align the health check with the documented endpoint. The team then confirms that the endpoint does not perform expensive database queries and sets a startup period suitable for application initialization.

The outcome is that the container remains healthy and the proxy stops removing it from service. The deeper lesson is that “unhealthy” identifies a failed test, not necessarily a failed process. Diagnosis required correlating health output, application logs, configuration, and network behaviour.

7. Illustrative Case Study: Checkout Service Incident

Background. An online retailer runs a checkout API and database in separate containers. Following a release, checkout requests become slow and some return errors.

Problem. The API container restarts several times, and the host reports low free disk space.

Analysis. Operators first preserve logs and events. Application logs show repeated database connection failures. docker stats shows moderate CPU but increasing block I/O. docker system df reveals that an uncontrolled debug logging configuration has filled the host's container log storage. The database container has also been configured with a volume, but the volume has little remaining capacity.

Approach. The team disables debug logging through a controlled configuration change, rotates or exports logs according to policy, and verifies available host and volume capacity. They then inspect database health and connection limits. A readiness check is added so the API does not receive traffic until its required dependencies are available. Resource alerts are created for disk and volume capacity, and log retention is defined.

Outcome. The API stops restarting, database connections become stable, and checkout latency returns to its baseline. The incident review identifies two contributing weaknesses: storage capacity was not monitored, and logging configuration had no production safeguard.

Learning points. Failures often cross boundaries. The visible symptom—slow checkout—was not explained by CPU usage alone. Evidence from events, logs, metrics, and storage inspection was required. Preventive operations included bounded logging, capacity alerts, dependency-aware readiness, and documented recovery procedures.

8. Operational Security, Governance, and Ethics

Operational data may contain personal information, tokens, customer identifiers, or commercially sensitive details. Logging and observability systems therefore require access control, encryption in transit and at rest, retention rules, and documented ownership. Debugging access should be proportional to the incident and audited.

Container images and runtime configurations should be scanned for vulnerabilities, unnecessary privileges, embedded secrets, and outdated base layers. Containers should run as non-root users where feasible, use read-only filesystems when appropriate, and receive only required capabilities. Troubleshooting should not normalize unsafe practices such as copying production secrets into shell commands or making unrestricted network changes.

Automated remediation, including restarts and scaling, must be governed carefully. A restart policy can improve availability, but repeated restarts may conceal a persistent defect or amplify load on a dependency. Alerts should be actionable and should distinguish a transient event from a sustained service-level problem.

9. Contemporary Practice and Best Practices

Modern container operations commonly combine Docker commands with centralized log aggregation, time-series metrics, distributed tracing, dashboards, alerting, and infrastructure-as-code. OpenTelemetry is increasingly used to standardize collection of traces, metrics, and logs, while platform environments may provide additional orchestration-level health and resource signals.

Artificial intelligence can assist with log clustering, anomaly detection, and probable-cause ranking. It should support—not replace—human verification. Models may misclassify rare failures, expose sensitive data, or recommend unsafe remediation. Production decisions require traceable evidence and appropriate approval.

Recommended practices are:

• Emit structured, timestamped, appropriately leveled logs.
• Monitor saturation as well as traffic, errors, and latency.
• Define health checks from service behaviour, not merely process existence.
• Set resource requests and limits from measured workloads.
• Separate persistent data from ephemeral container layers.
• Test DNS, ports, protocols, and authentication as distinct network layers.
• Preserve evidence before restarting or deleting resources.
• Use versioned configuration and reproducible images.
• Automate safe checks in CI/CD, but retain human review for destructive actions.
• Conduct post-incident reviews focused on learning rather than blame.

Challenges and Limitations

Observability creates cost in storage, network transfer, computation, and administration. Excessive telemetry can itself degrade a system and obscure important signals. Sampling can reduce cost but may miss rare events. Health checks can generate false positives, and metrics collected at coarse intervals may hide short resource spikes. Container-level visibility may also be incomplete when the real constraint is the host kernel, storage subsystem, cloud platform, or external dependency.

No single signal proves causation. A simultaneous rise in latency and CPU is correlation; controlled testing is needed to establish the cause. Troubleshooting remains a technical and organizational activity requiring system knowledge, communication, change control, and careful risk management.

Chapter Summary

Reliable container operations depend on observing both application behaviour and the runtime environment. Logs explain detailed events, metrics reveal numerical trends, events record lifecycle changes, health checks test service fitness, and traces connect work across components. Docker commands such as logs, events, inspect, stats, and network or volume inspection provide essential evidence.

Effective troubleshooting defines the symptom, establishes scope, preserves evidence, ranks hypotheses, tests narrowly, applies a safe correction, verifies recovery, and documents prevention. Build, startup, storage, networking, and performance failures require different diagnostic questions, but all benefit from layered analysis. Resource limits, health checks, logging policies, security controls, and capacity monitoring are operational design decisions rather than afterthoughts.

Key Terms

Observability: The ability to infer internal system state from externally produced evidence.

Monitoring: Continuous measurement and reporting against known conditions or thresholds.

Log: A recorded message describing an event or application activity.

Event: A record of a change in Docker object or runtime state.

Metric: A numerical, time-associated measurement.

Trace: A connected record of a request across services.

Span: A timed operation within a distributed trace.

Health check: An automated test of service condition.

Liveness: Whether a process should continue running or be restarted.

Readiness: Whether a service is prepared to receive traffic.

Exit code: A numeric status returned when a process terminates.

Container runtime: Software that creates and manages running containers.

Writable layer: Ephemeral filesystem space associated with a container.

Volume: Persistent or separately managed storage mounted into a container.

Port publishing: Mapping a host port to a container port.

Resource limit: A maximum permitted use of a resource such as CPU or memory.

Structured logging: Logging in a machine-readable format with consistent fields.

Restart storm: Repeated automated restarts that fail to resolve an underlying problem.

Readiness gate: A condition that must be satisfied before traffic is accepted.

Incident review: A structured examination of failure, response, causes, and improvements.

Review Questions

5 Short Answer Questions

1. What is the difference between a Docker event and a container log?
2. Why can a running process still be considered unhealthy?
3. What information does docker stats provide?
4. Why should persistent application data usually not be stored only in a container writable layer?
5. What does port publishing accomplish?

5 Long Answer Questions

1. Explain how logs, metrics, events, health checks, and traces complement one another in container operations.
2. Describe a systematic troubleshooting workflow and explain why evidence should be preserved before restarting a container.
3. Discuss the main causes of container startup failure and the commands that can help diagnose them.
4. Explain how storage and networking failures can produce apparently similar application symptoms.
5. Evaluate the benefits and limitations of automated health checks and restart policies.

3 Analytical / Application Questions

1. A web container is running, but clients receive connection-refused errors. Develop a layered diagnostic plan that distinguishes process, interface, container network, port publishing, and client-resolution problems.
2. A service has low average latency but an unacceptable p99 latency and occasional memory termination. Identify the evidence you would collect and propose a sequence of tests and corrective actions.
3. During an incident, logs contain customer email addresses and access tokens. Explain the immediate operational response and design a compliant long-term logging and access-control policy.

Multiple Choice Questions with Answers

Question 1

Which signal primarily records changes such as container creation, stopping, and health-status transitions?

A. Metrics

B. Events

C. Traces

D. Image layers

Correct Answer: B. Events

Question 2

What is the principal purpose of a container health check?

A. To compress the image

B. To publish a host port

C. To test whether the service is functioning as intended

D. To allocate additional memory

Correct Answer: C. To test whether the service is functioning as intended

Question 3

Which command follows new log output while showing only the most recent 100 lines initially?

A. docker logs --follow --tail 100 my-container

B. docker stats --tail 100 my-container

C. docker events --follow 100 my-container

D. docker inspect --logs 100 my-container

Correct Answer: A. docker logs --follow --tail 100 my-container

Question 4

Why should persistent data generally be stored in a volume or external data service?

A. Container writable layers are always encrypted automatically

B. Container writable layers are ephemeral and may disappear when containers are replaced

C. Volumes prevent all application errors

D. External services eliminate the need for backups

Correct Answer: B. Container writable layers are ephemeral and may disappear when containers are replaced

Question 5

What does EXPOSE in a Dockerfile do?

A. It automatically publishes the port to every host interface

B. It documents an intended container port

C. It creates a persistent volume

D. It limits CPU usage

Correct Answer: B. It documents an intended container port

Question 6

A service reports 20 failed requests out of 500 total requests in the same interval. What is its error rate?

A. 2%

B. 4%

C. 20%

D. 25%

Correct Answer: B. 4%

Question 7

Which diagnostic sequence best isolates a network failure?

A. Delete the image, then rebuild without inspection

B. Check listening process, network attachment, name resolution, connection, protocol, and authentication

C. Increase memory until the connection succeeds

D. Disable all health checks permanently

Correct Answer: B. Check listening process, network attachment, name resolution, connection, protocol, and authentication

Question 8

What is a major risk of including every dependency in a liveness check?

A. The image becomes smaller

B. The service may enter a restart storm during a temporary dependency outage

C. Logs become structured automatically

D. CPU limits are removed

Correct Answer: B. The service may enter a restart storm during a temporary dependency outage

Question 9

Which command is most directly useful for viewing current container CPU and memory consumption?

A. docker stats

B. docker port

C. docker volume ls

D. docker image history

Correct Answer: A. docker stats

Question 10

Why are p95 or p99 latency values often useful in addition to average latency?

A. They reveal behaviour among slower requests that an average may conceal

B. They measure image size instead of latency

C. They guarantee that no request fails

D. They eliminate the need for logs

Correct Answer: A. They reveal behaviour among slower requests that an average may conceal