Persistent Data, Volumes and Configuration

Introduction

Containers are intentionally ephemeral. A container can be stopped, replaced, or recreated from its image without preserving changes made during execution. This characteristic supports reproducible deployments, but it creates a central operational question: where should data that must outlive a container be stored?

Docker provides several storage mechanisms: the container’s writable layer, bind mounts, named volumes, and tmpfs mounts. These mechanisms differ in ownership, portability, performance, security, backup procedures, and lifecycle management. Correct selection is especially important for databases, uploaded files, logs, certificates, application configuration, and credentials.

This chapter examines persistent data and configuration management in Docker-based systems. It also considers backup and restoration, secrets, stateful-container risks, and professional governance practices.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain Docker’s layered storage model and writable container layers.
2. Compare bind mounts, named volumes, and tmpfs mounts.
3. Select appropriate storage for application data, configuration, cache, and secrets.
4. Create, inspect, populate, back up, and restore named volumes.
5. Distinguish configuration from secrets and apply safer injection methods.
6. Identify operational risks associated with stateful containers.
7. Design a practical persistence strategy for a multi-container application.

1. The Container Storage Model

A Docker image consists of read-only filesystem layers. Each instruction that changes the filesystem during image construction may contribute a layer. When Docker starts a container, it adds a container-specific writable layer above those image layers.

The union filesystem presents these layers as one apparent directory tree. A process inside the container normally sees paths such as /app, /etc, and /var/lib, without needing to know which layer contains each file.

When a process modifies a file originating in a read-only image layer, Docker commonly uses a copy-on-write mechanism: the file is copied into the writable layer and modified there. New files are also created in that layer. The original image remains unchanged.

The writable layer belongs to the individual container, not to the image. If the container is removed, data stored only in this layer is normally removed with it. Creating a new container from the same image does not recreate runtime changes.

1.1 What belongs in the writable layer?

The writable layer is suitable for:

• temporary application files;
• short-lived process output;
• experimentation during development;
• data whose loss is acceptable when the container is replaced.

It is generally unsuitable for:

• database files;
• user uploads;
• business records;
• certificates that must survive replacement;
• backups or audit records;
• large, frequently changing datasets.

A writable layer can also introduce performance and observability complications. Copy-on-write may impose overhead, and storage consumption is tied to the container rather than managed as an independently visible data resource.

2. Storage Choices in Docker

Docker’s main storage choices are compared below.

The choice should be based on data lifetime, sensitivity, access requirements, portability, performance, and recovery objectives rather than convenience alone.

2.1 Bind mounts

A bind mount maps a specific host path into a container. For example:

docker run --rm \
  --mount type=bind,src="$PWD/config",dst=/etc/myapp:ro \
  myapp:1.0

The host directory is the source of truth. Changes made by the host or the container, if the mount is writable, are visible through the mount.

Bind mounts are valuable when:

• developers need live source-code synchronization;
• a container must read a host-managed configuration file;
• an existing host directory must be exposed;
• integration with a host service is intentional.

They have important disadvantages. A path may not exist on another machine, permissions may differ between hosts, and a container may accidentally modify sensitive host files. Relative assumptions about host paths reduce deployment portability. Read-only mode, expressed with :ro or readonly, should be used whenever writing is unnecessary.

2.2 Named volumes

A named volume is created and managed through Docker. Its data is stored outside the container’s writable layer and remains available when the container is removed.

docker volume create app-db-data

docker run -d --name database \
  --mount source=app-db-data,target=/var/lib/postgresql/data \
  postgres:16

Named volumes are usually the default choice for container-managed persistent application data. They avoid hard-coding a host path and can be attached to replacement containers.

Useful commands include:

docker volume ls
docker volume inspect app-db-data
docker volume rm app-db-data

Removing a volume is destructive. A volume should not be deleted merely because its container has been removed. In production, deletion should be controlled through backup verification, change approval, and an explicit retention policy.

A named volume is not automatically a backup. It is persistent storage, not a second copy. If the host fails, the filesystem is corrupted, or an administrator deletes the volume, persistence alone does not provide recovery.

2.3 tmpfs mounts

A tmpfs mount stores files in volatile memory rather than in persistent disk storage:

docker run --rm \
  --mount type=tmpfs,dst=/run/secrets,tmpfs-size=1m \
  myapp:1.0

The contents disappear when the container stops and are not intended for durable records. tmpfs is useful for temporary credentials, session material, scratch files, and high-speed caches. It can reduce the chance that sensitive temporary data remains on disk, although it does not eliminate all risks: privileged users, debugging tools, process memory, swap configuration, and host access must also be considered.

Because tmpfs consumes memory, excessive use can create memory pressure. A size limit should be considered where supported and appropriate.

2.4 Read-only root filesystems

A container can often run with a read-only root filesystem:

docker run --read-only \
  --mount source=app-cache,target=/var/cache/myapp \
  myapp:1.0

The application can write only to explicitly mounted writable locations. This reduces accidental modification and limits some classes of persistence and tampering. It requires the application’s writable paths to be understood in advance. Common writable locations include a cache, a runtime socket directory, or a data directory.

3. Selecting Storage by Data Classification

Storage design begins with data classification. A useful classification is:

The key question is not simply whether data should persist. It is whether data should persist across container restarts, container replacement, host replacement, deployment rollback, or disaster recovery. These are different requirements.

A useful operational distinction is:

• Persistence: data survives an ordinary container replacement.
• Backup: a separate recoverable copy exists.
• Replication: data exists concurrently in more than one location.
• Recovery: a tested process can restore service and data.

A volume may provide the first property without providing the other three.

4. Volume Lifecycle and Initialization

Many official images initialize a data directory when it is empty. For example, a database image may create system tables and default configuration on first startup. Attaching an existing volume can therefore cause different behavior from attaching a new volume.

A safe lifecycle normally includes:

1. Create or select a volume with a documented purpose.
2. Attach it to the correct path expected by the image.
3. Initialize it once under controlled conditions.
4. Record ownership, permissions, image version, and schema version.
5. Back it up before upgrades or migrations.
6. Remove it only after retention and recovery requirements are satisfied.

Do not assume that copying arbitrary database files while the database is running produces a consistent backup. Application-aware backups, such as database dump or snapshot procedures, are often safer because they coordinate with transactions and internal state.

5. Backing Up and Restoring a Named Volume

For file-oriented data, a temporary container can archive a volume. The following example writes an archive to the current host directory:

docker run --rm \
  --mount source=app-data,target=/data,readonly \
  --mount type=bind,source="$PWD",target=/backup \
  alpine:3.20 \
  tar czf /backup/app-data.tgz -C /data .

The volume is mounted read-only to reduce accidental modification. The archive must then be protected, tested, and retained according to policy.

A basic restoration pattern is:

docker volume create app-data-restored

docker run --rm \
  --mount source=app-data-restored,target=/data \
  --mount type=bind,source="$PWD",target=/backup,readonly \
  alpine:3.20 \
  sh -c 'tar xzf /backup/app-data.tgz -C /data'

Restoration should normally occur into a new volume first. This preserves the original volume for comparison and rollback. File ownership and permissions must be checked because archive extraction performed as a different user can produce unusable files.

5.1 Backup quality

A professional backup process addresses:

• Consistency: Was the data captured in a valid application state?
• Completeness: Were all required files, metadata, and schemas included?
• Confidentiality: Is the backup encrypted and access-controlled?
• Integrity: Can corruption or tampering be detected?
• Retention: How many historical copies are required?
• Recovery time objective (RTO): How quickly must service be restored?
• Recovery point objective (RPO): How much recent data loss is acceptable?
• Testing: Has restoration been performed successfully?

A backup that has never been restored in a test is an assumption, not evidence of recoverability. Backup logs should avoid exposing passwords, tokens, or sensitive records.

6. Configuration Management

Configuration determines how an application behaves in a particular environment. Examples include database hostnames, listening ports, log levels, feature flags, and connection timeouts.

Configuration should be externalized from the image when it varies by environment. Common mechanisms include environment variables, mounted files, command-line arguments, and configuration services.

Example:

docker run --rm \
  --env-file production.env \
  --mount type=bind,src="$PWD/config/app.yml",dst=/etc/myapp/app.yml,readonly \
  myapp:1.0

Environment files improve repeatability but must be protected if they contain sensitive values. Configuration should be versioned when appropriate, reviewed, validated, and separated into development, test, and production variants.

A robust application should fail clearly when required configuration is absent or invalid. Silent fallback to insecure defaults is dangerous. Configuration schemas, startup validation, and documented precedence rules reduce deployment errors.

6.1 Configuration versus secrets

A database hostname is normally configuration. A database password is a secret. The distinction is based on sensitivity, not merely on the variable name.

Secrets should not be:

• embedded in Dockerfiles;
• committed to source control;
• placed in public image layers;
• printed in logs;
• exposed through diagnostic endpoints;
• passed unnecessarily to unrelated containers.

Docker Compose supports secret-oriented declarations, and orchestrators such as Swarm and Kubernetes provide more advanced secret mechanisms. In larger environments, a dedicated secret manager may provide encryption, access policies, auditing, rotation, and short-lived credentials.

A secret mounted as a file is not automatically safe. File permissions, container privileges, host access, backups, crash dumps, and application logging remain relevant. Secret rotation also requires an application design that can reload or safely restart with new credentials.

7. Stateful Containers and Distributed Systems Risks

Containers are easiest to replace when they are stateless. A stateless service can usually be recreated from an image and configuration. A stateful service owns data whose identity, consistency, and recovery must be managed.

Common risks include:

7.1 Single-host dependency

A local volume is generally tied to a Docker host unless a storage driver or external storage system provides mobility. Host failure can therefore make the container and its data unavailable together.

7.2 Split-brain and duplicate writers

Starting two containers that write to the same database directory is usually unsafe. Filesystem-level sharing does not create database coordination. Applications require transaction protocols, locking semantics, and a supported replication design.

7.3 Identity and permissions

The user inside a container may have a numeric user ID that differs from the host user. A bind-mounted directory can consequently become unreadable or writable by the wrong account. Images and deployment definitions should document expected ownership and avoid running with unnecessary root privileges.

7.4 Upgrade and rollback hazards

Database schema migrations are often not backward-compatible. Rolling back the image without rolling back the schema can leave the application unusable. Upgrades require tested migration plans, backups, compatibility analysis, and a rollback strategy.

7.5 Resource exhaustion

A container can fill a volume or host filesystem. Database growth, uncontrolled logs, and temporary files require quotas, monitoring, retention, and alerting. Log rotation must be designed separately from application data retention.

7.6 Security exposure

A writable bind mount can expose host files. A mounted Docker socket can provide near-host-level control and should be treated as highly privileged. Containers that can inspect or modify storage belonging to other workloads violate isolation expectations.

8. Applied Case: An Illustrative Online Bookstore

Background

An online bookstore consists of a web service, an API service, a PostgreSQL database, and an image-processing worker. Customers upload book-cover images, while the database stores orders and catalogue information.

Problem

The initial deployment stores database files and uploaded images in container writable layers. After a routine deployment, the containers are recreated and customer uploads disappear. Developers then propose bind-mounting the entire host filesystem to prevent future loss.

Analysis

The first design confuses container replacement with data retention. The proposed host-wide bind mount would create excessive privilege and portability risks. The data has different characteristics and should not share one storage mechanism.

Approach and solution

The team adopts the following design:

The database is backed up using a database-consistent method, and volume-level archives are used only where appropriate. Restoration is tested in an isolated environment. The API validates required configuration at startup, and containers run with non-root users where supported.

Outcome

A deployment can replace web and worker containers without losing data. Database recovery is documented and tested. The storage design is more portable than a host-wide bind mount and exposes fewer host resources.

Learning points

1. Different data types deserve different lifecycles.
2. A named volume improves persistence but does not replace backup.
3. A bind mount should be narrow, intentional, and preferably read-only.
4. Stateful services require migration, recovery, and capacity planning.
5. Secrets need controls beyond ordinary configuration management.

9. Professional Practices and Governance

A production storage design should be documented as an operational contract. At minimum, document each mount’s source, destination, mode, owner, data classification, backup method, retention period, and deletion procedure.

Recommended practices include:

• prefer --mount syntax because its fields are explicit;
• use named volumes for Docker-managed durable data;
• use bind mounts only for deliberate host integration;
• mark mounts read-only whenever possible;
• use a read-only root filesystem where practical;
• run processes as non-root users;
• monitor volume and host capacity;
• encrypt sensitive backups and restrict access;
• test restoration regularly;
• pin or control image versions for stateful services;
• record schema and data migration procedures;
• avoid storing secrets in images, Git repositories, or logs;
• define retention and secure deletion requirements.

Governance may also be affected by privacy and regulatory obligations. Customer records, health information, financial data, and authentication material may require access logging, encryption, retention limits, regional storage controls, and secure disposal. The exact obligations depend on the organization’s jurisdiction and sector; technical teams should involve security, legal, and compliance specialists rather than assuming that a container boundary satisfies regulation.

10. Contemporary Developments

Modern platforms increasingly separate compute from durable storage. Containerized applications may use managed databases, object-storage systems, distributed filesystems, or CSI-compatible storage in Kubernetes. This can improve availability and operational maturity but introduces network dependency, cost, latency, and provider-governance considerations.

Ephemeral environments and infrastructure-as-code make disposable containers practical, but they do not make business data disposable. Automated deployments should provision storage, access policies, backups, and monitoring together. Policy-as-code and secret-scanning tools can prevent common mistakes before deployment.

AI-assisted development can help generate Docker Compose files, backup scripts, and configuration templates, but generated artifacts require human review. AI tools may reproduce insecure patterns such as plaintext secrets, unrestricted bind mounts, or destructive volume commands. They should not receive confidential production data unless approved controls exist.

Chapter Summary

Docker separates immutable image layers from a container-specific writable layer. The writable layer is appropriate for temporary data but is lost when the container is removed. Bind mounts expose explicit host paths and are useful for development or controlled host integration, but they create coupling and permission risks. Named volumes are Docker-managed resources suited to durable application data, although they are not backups. tmpfs mounts provide volatile, memory-backed storage for temporary or sensitive runtime data.

Effective persistence design classifies data by lifetime, sensitivity, consistency, and recovery requirements. Configuration should be externalized and validated; secrets should be protected through secret-specific controls. Stateful containers require attention to host dependency, permissions, concurrency, upgrades, capacity, backup, restoration, and security. A professional solution combines appropriate mounts with tested recovery procedures, monitoring, least privilege, and clear governance.

Key Terms

Writable layer: The container-specific filesystem layer where runtime changes are stored.

Image layer: A read-only filesystem layer forming part of a Docker image.

Copy-on-write: A mechanism that copies a lower-layer file before modifying it.

Bind mount: A mapping of a specific host path into a container.

Named volume: A Docker-managed persistent storage resource identified by name.

tmpfs: A volatile, memory-backed filesystem mount.

Read-only mount: A mount that allows the container to read but not modify the mounted data.

Persistence: Survival of data beyond the ordinary lifetime of a container.

Backup: A separate copy of data retained for recovery.

Restoration: The process of rebuilding usable data from a backup.

RPO: Recovery point objective; the acceptable amount of data loss measured in time.

RTO: Recovery time objective; the target time for restoring service.

Configuration: Non-secret values that control application behavior.

Secret: Sensitive information such as a password, token, or private key.

Stateful service: A service that owns or depends on durable application data.

Schema migration: A controlled change to a database’s structure.

Least privilege: Providing only the permissions required for a task.

Object storage: A storage service that manages data as objects rather than ordinary local files.

Review Questions

Short Answer Questions

1. Why is a container writable layer normally unsuitable for database storage?
2. State one important difference between a bind mount and a named volume.
3. What happens to data in a tmpfs mount when the container stops?
4. Why is a named volume not equivalent to a backup?
5. Give two examples of information that should be treated as secrets.

Long Answer Questions

1. Compare writable layers, bind mounts, named volumes, and tmpfs mounts in terms of lifecycle, portability, security, and typical use.
2. Describe a reliable backup and restoration process for a Dockerized database.
3. Explain how configuration should be externalized and validated in different deployment environments.
4. Discuss the operational risks of running stateful services in containers.
5. Explain why read-only root filesystems and least privilege improve container security.

Analytical / Application Questions

1. Design a storage strategy for a service that stores customer records, uploaded documents, temporary conversion files, logs, and database credentials. Justify each choice.
2. A team wants to mount / from the host into a container so that no files are lost. Analyze the security, portability, and operational problems and propose a safer design.
3. A database container is upgraded and its migration succeeds, but the new image fails and the team wants to roll back. Develop a recovery plan that addresses schema compatibility, backups, volume handling, and service restoration.

Multiple Choice Questions with Answers

Question 1

Which storage mechanism is normally most appropriate for Docker-managed database data?

A. The container writable layer

B. A named volume

C. A host-wide bind mount

D. A temporary tmpfs mount

Correct Answer: B. A named volume

Question 2

What is the defining characteristic of a bind mount?

A. It is stored only in container memory

B. It is automatically encrypted by Docker

C. It maps a specific host path into the container

D. It disappears whenever the container stops

Correct Answer: C. It maps a specific host path into the container

Question 3

Which statement about a named volume is correct?

A. It is automatically a disaster-recovery backup

B. It normally survives removal of the container using it

C. It can only be used for read-only data

D. It must always refer to a path in the application image

Correct Answer: B. It normally survives removal of the container using it

Question 4

Which use is most appropriate for a tmpfs mount?

A. Permanent customer order records

B. A database’s primary data directory

C. Temporary processing files

D. Long-term backup archives

Correct Answer: C. Temporary processing files

Question 5

Why should a mount be marked read-only when writing is unnecessary?

A. To increase the image size

B. To reduce accidental modification and tampering

C. To make the container stateful

D. To ensure the data is deleted immediately

Correct Answer: B. To reduce accidental modification and tampering

Question 6

Which item is normally a secret rather than ordinary configuration?

A. Application listening port

B. Log verbosity level

C. Database password

D. Feature flag name

Correct Answer: C. Database password

Question 7

What is the principal purpose of testing restoration from a backup?

A. To increase the container’s writable layer

B. To prove that the backup can produce usable recovered data

C. To replace access control policies

D. To remove the need for replication

Correct Answer: B. To prove that the backup can produce usable recovered data

Question 8

What is a major risk of storing persistent data only on a local Docker host?

A. The image becomes read-only

B. The data may be unavailable if that host fails

C. The container cannot expose ports

D. The application cannot use environment variables

Correct Answer: B. The data may be unavailable if that host fails

Question 9

Which practice best protects a database during a version upgrade?

A. Delete the existing volume before upgrading

B. Run two writers against the same database directory

C. Create and test a recovery plan before migration

D. Embed the database password in the Dockerfile

Correct Answer: C. Create and test a recovery plan before migration

Question 10

What does RPO describe?

A. The target time required to restore service

B. The acceptable amount of data loss measured in time

C. The number of containers in a deployment

D. The maximum size of a named volume

Correct Answer: B. The acceptable amount of data loss measured in time