Cloud Storage, Databases and Data Protection

Introduction and Context

Cloud applications depend on persistent data. Compute instances may be replaced, containers may be rescheduled, and serverless functions may be recreated, but business data must remain available, accurate, secure and recoverable. Cloud storage and database services provide this persistence through different models, each designed for particular access patterns and operational requirements.

This chapter compares object, block and file storage; relational, NoSQL and managed database services; and the principal techniques used to protect data. It focuses on AWS and Google Cloud Platform (GCP), while emphasising concepts that apply across providers. The central engineering task is not simply choosing a service. It is matching data characteristics, workload behaviour, compliance obligations, recovery objectives and operational capability to an appropriate design.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish object, block and file storage;
• select suitable AWS and GCP storage services for common workloads;
• compare relational, NoSQL and managed database approaches;
• design lifecycle rules for cost and retention management;
• explain encryption, key management, backup and replication;
• define and apply recovery point objectives and recovery time objectives;
• identify operational, security, governance and compliance risks; and
• evaluate a practical cloud data architecture.

1. Storage Models and Their Uses

1.1 Object storage

Object storage stores data as objects within containers called buckets. Each object consists of content, metadata and a unique key. Applications normally access objects through HTTP-based APIs rather than by mounting a traditional disk. Object storage is highly scalable and well suited to unstructured or semi-structured data.

Typical uses include:

• images, video, audio and documents;
• software packages and deployment artefacts;
• log archives and audit records;
• data lakes and analytical input files; and
• backups and long-term archives.

AWS Simple Storage Service (Amazon S3) and Google Cloud Storage are object-storage services. Both support multiple storage classes, access controls, encryption, versioning and lifecycle policies.

Object storage is not normally a replacement for a low-latency database or a boot disk. Although providers offer strong durability and high throughput, access usually involves API requests and object-level operations. Applications should also avoid treating an object key as a substitute for database transactions or relational constraints.

1.2 Block storage

Block storage presents virtual disk volumes to a compute instance. The operating system formats the volume with a file system, and applications read and write blocks through ordinary disk operations. Block storage is appropriate when software requires a disk-like interface, low latency or control over the file system.

AWS Elastic Block Store (EBS) and Google Persistent Disk are common examples. They are used for operating-system volumes, transactional database files, indexes and applications requiring persistent attached storage.

A block volume is generally associated with a particular zone and compute environment, although some provider features support snapshots, replication or multi-attachment under defined conditions. It should not automatically be assumed to be a shared file system. Data consistency, attachment rules and failover behaviour must be verified for the selected volume type.

1.3 File storage

File storage provides a hierarchical directory structure and file semantics. Multiple servers can mount the same file system, subject to the service's access and performance characteristics. It is useful for shared content, home directories, content-management systems and legacy applications that expect files and folders.

AWS Elastic File System (EFS) and Amazon FSx families, together with Google Filestore, provide managed file-storage options. File storage is convenient where applications require shared paths, but it may be more expensive or less scalable than object storage for large archives. It also introduces file locking, permissions, mount management and throughput considerations.

1.4 Comparative view

1.5 Choosing a storage model

Selection should begin with workload requirements rather than product names. Ask the following questions:

1. Does the application need a file-system interface, a disk interface or an API?
2. Are operations mainly sequential, random, transactional or archival?
3. Must several hosts access the same data concurrently?
4. What latency, throughput and consistency are required?
5. How frequently is the data read, and how quickly must it be restored?
6. What retention, residency and deletion requirements apply?

A common architecture uses all three models: object storage for uploaded media and backups, block storage for database volumes, and file storage for shared configuration or legacy content.

2. Database Service Models

2.1 Relational databases

Relational databases represent data in tables containing rows and columns. Relationships are expressed through keys, and schemas define data types and constraints. SQL supports querying, joining, aggregating and modifying data.

Relational systems are suitable when data has stable relationships and the application requires transactions. A transaction is commonly evaluated using ACID properties:

• Atomicity: all operations in a transaction occur, or none do.
• Consistency: committed transactions preserve declared rules and constraints.
• Isolation: concurrent transactions behave according to the selected isolation model.
• Durability: committed data survives an accepted failure scenario.

AWS services include Amazon RDS and Amazon Aurora. GCP provides Cloud SQL and AlloyDB. These services automate activities such as provisioning, patching, backups and some forms of high availability, but they do not eliminate database administration. Capacity planning, schema design, query performance, permissions and recovery testing remain customer responsibilities.

2.2 NoSQL databases

NoSQL is a broad category rather than one single data model. Important forms include:

• key-value stores, which retrieve values by keys;
• document databases, which store JSON-like documents;
• wide-column databases, which organise data by partition and clustering keys; and
• graph databases, which represent entities and relationships as vertices and edges.

NoSQL systems often prioritise horizontal scalability, flexible schemas or predictable access patterns. They may provide strong consistency, eventual consistency or configurable consistency; this must be checked for the particular service. Designing the data model around application queries is usually more important than beginning with a normalised relational schema.

AWS DynamoDB is a managed key-value and document database. Amazon DocumentDB provides a document-oriented interface for supported workloads. GCP services include Firestore, Bigtable and managed offerings from partners. Each has different indexing, transaction, consistency and scaling behaviour.

2.3 Managed database services

A managed service transfers selected operational tasks from the customer to the provider. The provider may operate hardware, storage replication, patching infrastructure and service control planes. The customer normally remains responsible for data classification, access policies, schema or application design, credentials, configuration and appropriate recovery procedures.

“Managed” does not mean “automatically secure” or “automatically recoverable.” A database can be highly available but still suffer logical corruption, accidental deletion or an incorrect deployment. Backups, point-in-time recovery and independent recovery tests are therefore essential.

2.4 Relational and NoSQL comparison

The choice is architectural, not ideological. A system may use a relational database for orders, a NoSQL store for high-volume session data, and object storage for documents.

3. Storage Lifecycle and Cost Management

Lifecycle management automates actions based on object age, storage class, tags, prefixes or version state. A typical policy may transition frequently accessed objects to an infrequent-access class after a defined period, archive older objects, and delete temporary data after its retention period.

A lifecycle design should specify:

• the data population to which the rule applies;
• the age or event that starts the transition;
• the destination storage class;
• minimum-storage-duration charges and retrieval charges;
• treatment of current and non-current object versions;
• legal holds and retention locks; and
• exclusions for data that must remain immediately available.

AWS S3 Lifecycle and GCP Cloud Storage Object Lifecycle Management implement comparable concepts, although names, timing and pricing differ. Policies should be tested against realistic object histories before production use.

Versioning protects against accidental overwrite or deletion by retaining previous object versions. It is not a complete backup: malicious or faulty automation may also delete versions, and storage costs can increase substantially. Object Lock or retention-policy mechanisms can provide stronger immutability where supported and correctly configured.

A lifecycle example is an application that stores customer reports. Current reports remain in a standard class. Reports older than 30 days move to an infrequent-access class, reports older than 180 days move to an archive class, and temporary processing files are deleted after 14 days. The policy must account for regulatory retention, retrieval latency and the provider's minimum storage periods before it is approved.

4. Encryption and Access Protection

Encryption reduces the value of exposed data but does not replace access control. It should be applied in transit and at rest.

• Encryption in transit uses protected network protocols such as TLS between clients, services and administrative interfaces.
• Encryption at rest protects stored objects, volumes, snapshots, database files and backups.
• Customer-managed keys provide additional control over key rotation, disabling and audit, but also create responsibility for key availability and permissions.

AWS Key Management Service (KMS) and Google Cloud Key Management Service (Cloud KMS) support managed cryptographic keys. Provider-managed encryption is often enabled by default, while customer-managed keys may be required by policy or compliance rules.

A sound key-management design separates duties. Application operators should not automatically have permission to destroy keys. Key use should be logged, rotation requirements documented, and recovery procedures should address the possibility of a disabled or inaccessible key. Encryption can make data unrecoverable if key material is deleted or permissions are incorrectly removed.

Access should follow least privilege. Use separate identities for applications, administrators and backup systems; prefer short-lived credentials; restrict access by resource, action and condition; and monitor unusual downloads, privilege changes and key usage. Public object access should be explicitly denied unless it is an intentional, reviewed requirement.

5. Backup, Replication and Recovery

5.1 Backup versus replication

A backup is a recoverable copy retained independently of the primary data path. Replication maintains one or more copies, often to improve availability or read performance. Replication alone is not sufficient protection against logical errors: an accidental deletion or corrupted update may be reproduced immediately on every replica.

Backups should be protected against accidental deletion, tested through restoration, and retained according to business and regulatory requirements. Important techniques include:

• full, incremental or continuous backups;
• point-in-time recovery using logs or change records;
• cross-zone or cross-region copies;
• immutable or write-once retention;
• separate backup accounts or projects; and
• documented restoration runbooks.

5.2 Recovery objectives

The Recovery Point Objective (RPO) is the maximum acceptable age of data after recovery. An RPO of 15 minutes means the organisation accepts the possibility of losing up to 15 minutes of changes. The Recovery Time Objective (RTO) is the maximum acceptable time from a disruptive event until the service is restored to an agreed operating state.

These are business requirements, not merely technical settings. A lower RPO commonly requires more frequent change capture, replication or transaction logging. A lower RTO may require pre-provisioned infrastructure, replicated databases, automation and tested failover. Cost and complexity generally increase as objectives become more demanding, although the relationship is not a universal mathematical law.

A useful recovery design distinguishes:

• availability: continuing service during expected component failures;
• disaster recovery: restoring service after a major site, region or platform failure; and
• data recovery: restoring information after deletion, corruption or an application mistake.

5.3 Backup strategy

A practical strategy begins with data classification. Mission-critical transactional data may require point-in-time recovery, multi-zone deployment and cross-region backup copies. Development data may require less frequent snapshots and shorter retention. Public, reproducible artefacts may not need the same controls as regulated customer records.

Backup schedules should be expressed in business terms. For example, a payment database might use continuous log archiving, daily recovery points and a cross-region copy. A development database might use nightly snapshots retained for seven days. Both strategies must be validated by restoration tests.

The “3-2-1” principle is a useful baseline: maintain at least three copies, on at least two types of storage or media, with at least one copy separated from the primary environment. It is a guideline rather than a guarantee. Modern cloud designs may extend it with immutability, separate administrative control and cross-region protection.

5.4 Replication patterns

Common patterns include:

• synchronous replication, in which a write is acknowledged after required replicas receive it; this can reduce data loss but may increase latency and depend on network distance;
• asynchronous replication, in which the primary acknowledges before all replicas receive the change; this often improves performance but introduces a replication lag and possible data loss; and
• read replicas, which improve read capacity but may not be suitable as independent disaster-recovery copies.

Failover must be designed, not assumed. DNS changes, connection strings, credentials, application retries, split-brain prevention and data consistency all require testing.

6. Infrastructure as Code and Operational Practice

Storage, databases, keys, policies and backups should be defined through reviewed infrastructure as code where possible. A deployment should make lifecycle rules, encryption settings, network boundaries, backup schedules and retention explicit. Manual console changes create configuration drift and make audits and recovery more difficult.

A professional operating model includes:

1. data classification and ownership;
2. documented service-level objectives;
3. least-privilege identities;
4. central logging and alerting;
5. quota, capacity and cost monitoring;
6. patch and version management;
7. automated backup verification;
8. periodic restore and failover exercises; and
9. change approval for destructive operations.

Monitoring should cover storage growth, failed backups, replication lag, database latency, transaction errors, unusual access and approaching service quotas. Cost monitoring is also a reliability practice: uncontrolled growth can cause budget exhaustion and service disruption.

7. Applied Case: Illustrative Online Learning Platform

Background

Consider an illustrative institution that operates an online learning platform. Students upload assignments, instructors publish videos and grades, and the platform records enrolments and payments. The institution requires continuous normal operation during a zone failure and must restore critical services after a regional outage.

Problem

The original system stores uploaded files on a single server disk and uses one relational database instance. Backups run irregularly, files are not encrypted under a centrally controlled key, and the team has never performed a full restoration. An accidental deletion could affect both the primary data and its copied files.

Analysis

Assignment files and videos are large, immutable after publication in most cases, and accessed through downloads. They are therefore better suited to object storage than to database BLOB columns or a local disk. Enrolments, grades and payments require relationships, constraints and transactions, so a relational database is appropriate. Temporary processing output has a short useful life and should not be retained indefinitely.

The institution sets an RPO of 15 minutes for grades and payments and an RTO of two hours for critical services. Media can tolerate a longer recovery time because it can be reprocessed from source material. Regulatory and institutional policies require controlled retention and auditability.

Approach and solution

The design uses an object-storage bucket for assignments and media, with private access, TLS, server-side encryption and a customer-managed key. Uploads are performed using short-lived signed requests rather than broad student permissions. Versioning is enabled for important content, and an immutability mechanism is applied to records subject to formal retention.

A managed relational database stores users, courses, submissions, grades and payments. It is deployed for high availability within the region, with point-in-time recovery and encrypted snapshots. Transaction logs and scheduled backup copies are sent to a separate recovery environment in another region. Database credentials are stored in a secrets-management service rather than in source code.

Lifecycle rules move older media to a lower-cost class and eventually to archive storage, subject to retention and retrieval requirements. Temporary transcoding files are deleted after a short period. Database backups have a documented retention schedule, and the recovery environment has infrastructure-as-code definitions for rapid recreation.

Outcome

The platform gains separation between structured and unstructured data, stronger access control, predictable retention and a tested recovery process. A zone failure is handled by the managed database's high-availability design, while a regional event is addressed through cross-region backup restoration and redeployment. The institution does not claim zero data loss; instead, it documents the agreed RPO and tests whether the design meets it.

Learning points

The case demonstrates that data protection is a system of complementary controls. Object versioning, database replication, encryption, lifecycle automation and backups solve different problems. Recovery objectives must be agreed with stakeholders, implemented technically and validated through exercises.

8. Challenges, Limitations and Governance

Cloud services reduce infrastructure administration but introduce dependency on provider APIs, pricing models, regional availability and service-specific limits. Migration can be difficult when applications rely on proprietary database features or incompatible storage semantics. Egress charges and archive-retrieval costs may materially affect a recovery plan.

Data residency and privacy laws may restrict where personal data and backups are stored. Organisations should identify data owners, retention periods, lawful processing bases and deletion obligations. A backup may preserve personal data after the primary record has been deleted, so retention and erasure policies must address backup copies explicitly.

Auditable controls should record who accessed data, who changed policies, when keys were used and whether backups succeeded. Logs themselves may contain sensitive information and require controlled retention. Artificial intelligence can assist with anomaly detection, schema analysis and cost optimisation, but automated recommendations must not be allowed to delete data or change retention without review. AI systems also require careful handling of training data, personal information and generated database queries.

9. Contemporary Developments

Cloud platforms increasingly provide serverless databases, automated tiering, distributed SQL, global tables, confidential-computing options and policy-as-code controls. These developments can reduce operational effort, but they do not remove the need to understand consistency, failure modes, cost and portability.

Data lakehouse architectures combine object storage with transactional table formats and analytical engines. They can support large-scale analytics while preserving some data-management features traditionally associated with databases. However, governance, schema evolution, metadata quality and access control remain significant engineering concerns.

The current professional direction is toward automated, policy-driven data platforms: encryption by default, identity-centred access, immutable backups, continuous compliance checks and recovery exercises integrated into delivery pipelines.

Best-Practice Checklist

Before approving a cloud data design, confirm that:

• the storage model matches the access pattern;
• structured data has an appropriate database model;
• data owners and classifications are documented;
• encryption in transit and at rest is enabled;
• key ownership, rotation and recovery are defined;
• lifecycle rules include versions, legal holds and minimum charges;
• backups are independent enough to resist common failure modes;
• RPO and RTO are agreed and measurable;
• replication is not being mistaken for backup;
• access is least-privilege and auditable;
• costs, quotas and retrieval charges are monitored; and
• restoration and failover have been tested.

Chapter Summary

Cloud storage is commonly divided into object, block and file models. Object storage is suited to scalable unstructured content and archives; block storage provides disk-like volumes for operating systems and databases; and file storage provides shared hierarchical access. Relational databases support structured relationships and transactions, while NoSQL systems support specialised access patterns, flexible models and large-scale partitioning. Managed services reduce infrastructure work but do not transfer responsibility for data, permissions, configuration or recovery.

Lifecycle policies control storage transitions and deletion, while versioning and immutability address different forms of protection. Encryption, identity management and key governance protect confidentiality and control access. Backups provide recoverable copies; replication improves availability but does not replace backups. RPO expresses tolerable data loss, and RTO expresses tolerable restoration time. Effective cloud data protection combines technical controls, governance, monitoring, infrastructure as code and repeated recovery testing.

Key Terms

Object storage: Storage of content as objects accessed through keys and APIs.

Block storage: Disk-like storage presented as addressable blocks to a host.

File storage: Hierarchical storage accessed through files and directories.

Bucket: A logical container for objects.

Lifecycle policy: Automated rule that transitions or deletes stored data.

Versioning: Retention of previous object versions after updates or deletions.

Relational database: Database organised into related tables and queried commonly with SQL.

NoSQL: A family of non-relational database models including key-value, document and wide-column systems.

Transaction: A unit of database work governed by defined consistency rules.

ACID: Atomicity, consistency, isolation and durability properties of transactions.

Managed service: A provider-operated service in which selected infrastructure tasks are abstracted from the customer.

Encryption at rest: Protection of stored data using cryptographic transformation.

Key management: Creation, control, rotation, use and protection of encryption keys.

Backup: A recoverable copy retained for restoration.

Replication: Maintenance of additional copies, often for availability or performance.

RPO: Recovery Point Objective; the maximum acceptable age of recovered data.

RTO: Recovery Time Objective; the maximum acceptable restoration time.

Point-in-time recovery: Restoration to a selected time using backups and change records.

Immutability: Protection that prevents alteration or deletion during a defined period.

Least privilege: Granting only the permissions required for an identity or task.

Review Questions

Short Answer Questions

1. What is the principal difference between object storage and block storage?
2. Give two examples of workloads suitable for file storage.
3. What do the letters ACID represent?
4. Define Recovery Point Objective and Recovery Time Objective.
5. Why is replication not a complete replacement for backup?

Long Answer Questions

1. Compare object, block and file storage in terms of interface, performance considerations and common applications.
2. Explain the differences between relational and NoSQL databases and identify a suitable use case for each.
3. Describe how lifecycle policies, versioning and immutability can work together in an object-storage design.
4. Explain the shared-responsibility implications of using a managed database service.
5. Design a backup and replication strategy for a business-critical relational database.

Analytical / Application Questions

1. A media company stores frequently accessed videos for 60 days and must retain them for seven years. Propose storage classes, lifecycle transitions, access controls and recovery protections, explaining the trade-offs.
2. An online retailer requires an RPO of 15 minutes and an RTO of two hours for orders, but can tolerate a one-day recovery period for product images. Design a differentiated data architecture and justify the choices.
3. A team enables database replication but has no tested backups and stores encryption keys in the same administrative account as the database. Identify the risks and propose a prioritised remediation plan.

Multiple Choice Questions with Answers

Question 1

Which storage model is most appropriate for large media files accessed through an HTTP API?

A. Object storage

B. Block storage

C. Local memory

D. CPU cache

Correct Answer: A. Object storage

Question 2

Which requirement most strongly favours a relational database?

A. Unstructured files with infrequent retrieval

B. Strong relationships, constraints and multi-step transactions

C. A simple key-based session lookup only

D. Long-term archival of application logs

Correct Answer: B. Strong relationships, constraints and multi-step transactions

Question 3

What is the main purpose of a lifecycle rule?

A. To define database table relationships

B. To automatically transition or delete data according to conditions

C. To increase CPU frequency

D. To replace encryption keys with passwords

Correct Answer: B. To automatically transition or delete data according to conditions

Question 4

An RPO of 15 minutes means that the organisation accepts which possibility after recovery?

A. The service will be unavailable for 15 minutes

B. Up to 15 minutes of recent data changes may be lost

C. The database must be restored within 15 minutes

D. Backups must be retained for 15 minutes

Correct Answer: B. Up to 15 minutes of recent data changes may be lost

Question 5

Why is replication alone insufficient as a data-protection strategy?

A. Replicas can reproduce accidental deletion or corruption

B. Replication always prevents high availability

C. Replicas cannot contain any database data

D. Replication disables encryption automatically

Correct Answer: A. Replicas can reproduce accidental deletion or corruption

Question 6

Which control most directly protects stored data if an unauthorised person obtains the storage media?

A. Encryption at rest

B. DNS round-robin

C. Horizontal scaling

D. Database indexing

Correct Answer: A. Encryption at rest

Question 7

Which database model commonly stores JSON-like records and is often designed around application access patterns?

A. Document database

B. Block volume

C. Tape library

D. File allocation table

Correct Answer: A. Document database

Question 8

What is a key risk of deleting a customer-managed encryption key?

A. The protected data may become unrecoverable

B. The database automatically becomes relational

C. Object lifecycle rules become faster

D. Network latency is guaranteed to decrease

Correct Answer: A. The protected data may become unrecoverable

Question 9

Which practice provides the strongest evidence that a recovery plan works?

A. Reading the backup success message once

B. Performing and documenting restoration tests

C. Increasing the number of application servers

D. Changing the database name

Correct Answer: B. Performing and documenting restoration tests

Question 10

What does least privilege require?

A. Every administrator receives full access to every resource

B. Identities receive only the permissions needed for their tasks

C. All data is made publicly readable

D. Backups are stored without access logs

Correct Answer: B. Identities receive only the permissions needed for their tasks