Cloud Storage and Managed Data Services

Introduction

Cloud applications depend on reliable, scalable, and appropriately governed data services. Unlike traditional infrastructure, where an organisation purchases and manages storage arrays and database servers, cloud platforms provide storage and data capabilities as services. These services can be provisioned rapidly, expanded elastically, replicated across locations, and integrated with identity, monitoring, backup, and automation systems.

The central design challenge is not simply choosing the largest or fastest storage service. It is matching the data access pattern, performance requirement, durability target, consistency model, recovery objective, security obligation, and cost constraint to an appropriate service. A well-designed solution may use object storage for documents, block storage for virtual-machine disks, file storage for shared directories, a managed relational database for transactions, a NoSQL database for high-volume key-value access, and a cache for frequently requested data.

This chapter examines these services and the decisions required to configure and operate them professionally.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish object, block, and file storage;
• select storage according to access patterns and performance requirements;
• explain durability, availability, replication, consistency, backup, and recovery;
• evaluate managed relational, NoSQL, analytical, and caching services;
• apply lifecycle and cost-management policies;
• calculate basic capacity, recovery, and availability measures;
• design a storage and data architecture for a realistic cloud application;
• identify security, governance, compliance, and operational risks.

1. Storage Abstractions in Cloud Computing

1.1 Object storage

Object storage stores complete objects together with metadata and a unique key. An object commonly consists of a data payload, a key or name, user-defined metadata, and system-managed attributes. Applications access objects through an API, usually over HTTP or HTTPS, rather than by modifying individual disk sectors.

Object storage is suitable for images, video, backups, software packages, logs, documents, datasets, and static website content. It provides very high aggregate capacity and is generally designed for exceptional durability. However, it is not normally a replacement for a low-latency transactional disk. Applications should treat an object as a complete unit; partial in-place updates are generally unavailable or inefficient.

Important object-storage concepts include:

• Bucket or container: a logical namespace holding objects.
• Key: the object’s unique name within a namespace.
• Metadata: descriptive information such as content type, owner, or retention status.
• Versioning: preservation of earlier object versions after updates or deletions.
• Storage class: a service tier optimised for access frequency, latency, or price.
• Multipart upload: division of a large object into independently uploaded parts.

Object storage is often eventually consistent in some operations on older or specialised systems, although many major cloud services now provide strong read-after-write behaviour for ordinary object operations. Designers must consult the chosen provider’s current documentation rather than assuming that all object stores have the same semantics.

1.2 Block storage

Block storage presents raw, fixed-size blocks to a host operating system. The operating system formats the volume with a file system such as ext4 or NTFS and manages files, directories, permissions, and allocation. A block volume may be attached to a virtual machine or, in some platforms, exposed to container workloads.

Block storage is appropriate for operating-system disks, transactional databases, enterprise applications, and workloads requiring predictable random input/output operations per second (IOPS). It often supports snapshots, encryption, resizing, provisioned IOPS, and attachment controls.

Block storage is normally associated with one host at a time unless a cluster-aware file system and suitable shared-access protocol are used. Simply attaching the same volume to several independent hosts can cause corruption because each host may cache and update file-system structures without coordination.

1.3 File storage

File storage exposes a hierarchical file system containing directories and files. It is commonly accessed through protocols such as NFS or SMB. Multiple compute instances can mount the same file system, making it useful for shared content, user home directories, content-management systems, and applications designed around shared file paths.

File storage is easier to adopt for applications that expect a conventional file system, but its performance, locking behaviour, throughput, and metadata scalability depend on the protocol and service configuration. It may be more expensive than object storage for large archives and less suitable than block storage for high-performance database volumes.

1.4 Comparison of storage types

2. Performance, Capacity, and Service Selection

Storage selection begins with workload characterisation. Important questions include:

1. How large are individual items?
2. Are reads sequential, random, or whole-object?
3. How many operations occur per second?
4. What latency is acceptable?
5. Are several hosts required to access the same data?
6. Does the application require transactions, range reads, or atomic updates?
7. How rapidly will data grow?
8. What are the recovery and retention requirements?

A useful first estimate of storage capacity is:

Here, N is the number of stored items, S is the average item size, O is the fractional overhead for metadata, indexes, and reserved space, C_versions is capacity consumed by retained versions, and C_backups is backup capacity. All capacity terms must use the same unit, such as GiB. This is an estimate, not a provider billing formula; compression, deduplication, replication, and minimum billable object sizes may change actual consumption.

For a workload generating R operations per second, average throughput can be estimated as:

where T is throughput in bytes per second and B is average bytes transferred per operation. This relationship is useful for rough sizing, but it does not establish latency or guarantee that a service can sustain the workload. Queue depth, operation size, concurrency, network limits, and provider quotas must also be tested.

For block storage, performance is often described using IOPS and throughput. Small random operations may be IOPS-limited, while large sequential transfers may be throughput-limited. A database requiring 8,000 random operations per second should not be sized only by volume capacity; its provisioned IOPS, latency, queueing, and host limits must also be considered.

3. Durability, Availability, Replication, and Consistency

3.1 Durability and availability

Durability is the probability that data will not be lost over a specified period. Availability is the proportion of time that a service or data access operation is usable. A service can have high durability but temporary unavailability, or high availability while still requiring backups to protect against deletion or corruption.

If independent failure assumptions are reasonable, the probability that at least one of n replicas remains available, where each replica has availability a, is:

This is a simplified model. It assumes independent failures, equal replica availability, and no common failure domain. Real cloud systems have correlated failures, network partitions, software defects, and operational errors, so provider service-level objectives and architecture documentation remain essential.

Replication does not automatically protect against every threat. Synchronous replication may copy accidental deletion immediately. Asynchronous replication may have a replication lag, causing a recovery point that is behind the primary. Backups, immutable retention, and tested restoration are therefore distinct controls.

3.2 Replication scope

Replication may occur:

• within one availability zone or data centre;
• across availability zones in one region;
• across geographically separate regions;
• across accounts, subscriptions, or projects;
• to an independent provider or offline medium.

A broader replication scope generally improves resilience to local failure but increases cost, latency, data-transfer charges, and governance complexity. Cross-border replication may also create legal and regulatory issues.

3.3 Consistency models

Consistency describes what a reader is allowed to observe after a write. Common models include:

• Strong consistency: after a successful write, subsequent reads observe the latest committed value, subject to the service’s defined scope.
• Eventual consistency: replicas may temporarily disagree, but converge if no further updates occur.
• Read-after-write consistency: a writer or specified reader sees a newly written value immediately, while other operations may have different guarantees.
• Session consistency: a client’s reads observe its own writes within a session.
• Causal consistency: operations that have a causal relationship are observed in an order that respects that relationship.

Consistency is not the same as durability. A strongly consistent system can faithfully and durably preserve an incorrect update. Applications must use validation, transactions, version checks, and authorisation to protect correctness.

4. Lifecycle Management and Cost Optimisation

Cloud storage commonly offers hot, cool, infrequent-access, archive, and deep-archive tiers. Hot tiers provide rapid access at a higher storage price; archive tiers reduce storage cost but may impose retrieval fees, minimum retention periods, and restoration delays.

A lifecycle policy automatically transitions or deletes data according to age, tags, prefix, legal hold, or object version. A typical policy might retain current operational data in a hot tier for 30 days, move it to an infrequent-access tier after 30 days, archive it after 180 days, and delete it after seven years unless a legal hold applies.

Lifecycle design must account for:

• access frequency and retrieval latency;
• minimum billable storage duration;
• retrieval and request charges;
• versioned objects and delete markers;
• backup and legal-retention requirements;
• encryption-key retention;
• compliance and e-discovery obligations.

A lifecycle rule must never be allowed to delete data still required for recovery or legal evidence. Test policies in a non-production environment and monitor transitions before enabling destructive deletion.

5. Managed Data Services

5.1 Managed relational databases

A managed relational database provides a database engine while the cloud provider operates much of the underlying infrastructure. The customer usually remains responsible for schema design, indexes, queries, transactions, access control, data classification, and application correctness.

Relational systems organise data into tables and support structured query languages, constraints, joins, and transactions. They are suitable for banking records, orders, inventory, billing, and other workloads requiring strong integrity.

A transaction commonly follows ACID principles:

• Atomicity: all operations in a transaction succeed or none take effect.
• Consistency: constraints and rules remain satisfied after commit.
• Isolation: concurrent transactions do not produce prohibited intermediate effects.
• Durability: committed data survives the failures covered by the service design.

Managed database features may include automated backups, point-in-time recovery, read replicas, multi-zone failover, encryption, patching, monitoring, and parameter management. These features reduce operational burden but do not eliminate the need for capacity planning, query optimisation, schema migration control, or restore testing.

5.2 NoSQL databases

NoSQL services are designed for specific access patterns and large-scale distribution. Key-value stores retrieve a value by a key; document stores manage semi-structured records; wide-column stores support partitioned, high-throughput workloads; graph databases represent relationships between entities.

NoSQL design usually begins with query patterns rather than normalised relational entities. Partition-key selection is critical. A poor key can create a hot partition, concentrating traffic on one shard and limiting scalability. Denormalisation and duplicated data may be acceptable when they reduce read complexity, provided update procedures and consistency implications are understood.

NoSQL systems may offer configurable consistency, conditional writes, time-to-live attributes, streams, and automatic scaling. Their apparent simplicity can conceal complexity in data modelling, secondary indexes, cross-record transactions, and reporting.

5.3 Data warehouses and data lakes

A data lake stores large volumes of raw or lightly processed data, frequently in object storage. A data warehouse structures data for analytical queries, reporting, and business intelligence. Modern architectures often combine a lake with catalogue, governance, transformation, and query services.

Operational databases should not automatically be used for intensive analytical queries. Separating transactional and analytical workloads protects production latency and allows each system to be optimised for its purpose.

5.4 Managed caches

A cache stores frequently used data closer to the application. In-memory caches can reduce database load and improve response time. Common patterns include:

• Cache-aside: the application reads the cache first; on a miss, it reads the database and populates the cache.
• Write-through: writes update the cache and backing store through a coordinated path.
• Write-behind: the cache acknowledges a write before asynchronously persisting it.
• Read-through: the cache service obtains missing data from the backing store.

Cache-aside is widely used, but invalidation is difficult. A cached value may become stale, and a failure between database update and cache deletion can produce inconsistent reads. Time-to-live values, versioned keys, explicit invalidation, and tolerable-staleness analysis are useful controls. A cache should not be the only durable copy of important data unless the service explicitly provides durable persistence and the design accepts its failure modes.

6. Backup, Recovery, and Business Requirements

Two core recovery objectives are:

• Recovery Point Objective (RPO): the maximum acceptable amount of data loss measured in time.
• Recovery Time Objective (RTO): the maximum acceptable time to restore service.

If backups occur every τ hours and failures can occur uniformly between backups, the maximum possible data loss is approximately τ hours and the average loss is approximately τ/2 hours. The average is only an estimate; the design must satisfy the maximum permitted loss.

Recovery planning should specify:

1. what data and configuration are protected;
2. backup frequency and retention;
3. encryption and key management;
4. geographic and account separation;
5. immutable or write-once retention;
6. restoration dependencies;
7. validation and reconciliation procedures;
8. recovery testing frequency and evidence.

A backup that has never been restored is an assumption, not assurance. Restoration tests should measure actual RTO, verify data integrity, validate application configuration, and record unresolved gaps.

7. Security, Governance, and Compliance

Cloud data protection is a shared responsibility. Providers secure the infrastructure and service platform, while customers configure identities, permissions, encryption, networks, retention, and application controls.

Best practices include:

• least-privilege identity and access management;
• separate administrative and application roles;
• encryption in transit and at rest;
• customer-managed keys where required, with controlled rotation;
• private endpoints or restricted network paths;
• prevention of public exposure through policy controls;
• access logging and anomaly detection;
• classification of personal, financial, health, and confidential data;
• retention schedules and defensible deletion;
• separation of production, test, and development data;
• masking or tokenisation of sensitive data in non-production environments.

Encryption does not solve every governance problem. A compromised application with legitimate decryption permissions may still read protected data. Access design, monitoring, segregation of duties, and secure application development remain necessary.

8. Applied Case: Illustrative Online Learning Platform

Background

An illustrative college operates an online learning platform. It stores lecture videos, assignment submissions, course metadata, grades, and usage events. Students access the platform globally, while lecturers upload large files and administrators require reliable reporting.

Problem

The original system stores everything on a single virtual-machine disk. Storage is nearing capacity, backups are irregular, and a disk failure could interrupt teaching. Video delivery is slow during examination periods, and administrators run reports directly against the transactional database.

Analysis

The data has different access patterns. Videos and assignment files are large, immutable or versioned objects. Course and enrolment records require transactional integrity. Usage events are high-volume and suitable for analytical processing. Frequently accessed course metadata can be cached. The platform requires point-in-time recovery for grades and must preserve submitted assignments according to institutional retention policy.

Approach and solution

The redesigned architecture uses object storage for videos and submissions, with versioning, server-side encryption, restricted access, and lifecycle transitions for inactive recordings. A content delivery service distributes public course media through controlled, signed requests rather than exposing the storage container.

A managed relational database stores users, courses, enrolments, assessment definitions, grades, and submission metadata. Multi-zone deployment and automated point-in-time backups reduce recovery risk. Database read replicas or a reporting export prevent analytical queries from competing with transactional operations.

A managed key-value store receives high-volume viewing events, partitioned by course and time while avoiding a single hot partition. Events are periodically transformed into warehouse tables for reporting. A managed in-memory cache stores course catalogues and short-lived session data. Cache entries have explicit expiration and are never treated as the authoritative grade record.

The institution defines an RPO of 15 minutes for grades and an RTO of two hours for the learning platform. Backup schedules, cross-account copies, restore runbooks, and quarterly recovery exercises are established. Access is controlled using separate student, lecturer, service, and administrator roles. Audit logs are retained and protected from alteration.

Outcome

The architecture separates storage according to workload, improves media delivery, reduces database contention, and provides measurable recovery capabilities. It also introduces new responsibilities: lifecycle policies must be reviewed, cache staleness must be monitored, and object access must be continuously audited.

Learning points

The case demonstrates that cloud design is a composition of specialised services rather than a single storage choice. Data classification, access patterns, recovery objectives, and governance requirements should determine the architecture. Managed services reduce infrastructure administration, but accountability for configuration and data remains with the organisation.

9. Contemporary Developments and Professional Practice

Current cloud architectures increasingly use serverless databases, distributed SQL systems, lakehouse platforms, policy-as-code, confidential computing, and automated data-quality controls. Artificial intelligence is relevant when analysing logs, forecasting capacity, classifying data, detecting anomalous access, or generating operational recommendations. AI-generated recommendations must be validated because inaccurate classifications or automated deletion decisions can cause data loss and compliance violations.

Professional engineers should use infrastructure as code, automated policy checks, performance tests, cost budgets, observability dashboards, and documented runbooks. Service selection should be evidence-based: benchmark representative workloads, measure tail latency rather than only averages, test failure scenarios, and review provider limits and pricing conditions.

Challenges and Limitations

Cloud data services introduce several trade-offs:

• abstraction reduces hardware administration but can obscure failure modes;
• elasticity may increase cost if consumption is not governed;
• replication improves resilience but can create stale reads and regulatory complexity;
• managed databases simplify operations but may impose engine, version, or portability constraints;
• caches improve speed but add invalidation and consistency problems;
• lifecycle automation reduces cost but can cause irreversible deletion;
• multi-region design improves disaster resilience but increases latency and operational complexity;
• provider-specific APIs can create vendor lock-in.

No architecture is universally optimal. A defensible design records assumptions, requirements, alternatives, test evidence, and residual risks.

Best-Practice Checklist

Before approving a cloud storage or managed-data design, confirm that:

• each data class has an appropriate service and owner;
• capacity, throughput, latency, and growth assumptions are documented;
• durability and availability requirements are distinguished;
• replication scope matches the failure model;
• consistency requirements are explicit;
• backup restoration has been tested;
• RPO and RTO have measurable evidence;
• lifecycle and deletion policies respect retention and legal holds;
• encryption, identity, network, and audit controls are configured;
• costs include storage, requests, retrieval, transfer, replicas, backups, and operations;
• monitoring detects capacity, latency, failures, and unusual access;
• portability and provider lock-in have been considered.

Chapter Summary

Cloud storage is divided into object, block, and file abstractions, each suited to different access patterns. Object storage provides scalable API-based storage for unstructured content; block storage supplies host-attached volumes for operating systems and transactional workloads; and file storage provides shared hierarchical access through file protocols.

Managed databases shift much infrastructure administration to the provider but leave customers responsible for data modelling, security, queries, correctness, and recovery. Relational systems support structured transactions and integrity, whereas NoSQL systems support specialised high-scale access patterns. Caches improve performance but are normally not authoritative data stores.

Durability, availability, replication, consistency, backup, RPO, and RTO describe different properties and must not be treated as interchangeable. Lifecycle policies can reduce cost, but they require careful consideration of retrieval, retention, versioning, legal holds, and deletion risk. A professional design combines technical performance with governance, security, operational testing, and financial control.

Key Terms

Object storage: API-based storage of complete data objects and metadata.

Block storage: Raw volume storage presented to a host operating system.

File storage: Shared hierarchical storage accessed through file protocols.

Bucket: A logical container for objects.

IOPS: Input/output operations per second.

Throughput: Quantity of data transferred per unit of time.

Durability: Likelihood that stored data will not be lost over a defined period.

Availability: Proportion of time a service or operation is usable.

Replication: Maintaining copies of data in multiple locations or systems.

Consistency: Rules governing what values readers may observe after writes.

Versioning: Retaining earlier states of an object or record.

Lifecycle policy: Automated rule governing tier transitions, retention, or deletion.

Managed database: A database service in which the provider operates much of the underlying infrastructure.

ACID: Atomicity, consistency, isolation, and durability transaction properties.

NoSQL: A family of non-relational data stores optimised for particular access patterns.

Cache: Fast temporary storage for frequently accessed data.

RPO: Maximum acceptable period of data loss measured in time.

RTO: Maximum acceptable time to restore service.

Point-in-time recovery: Restoration of data to a selected moment within a backup window.

Immutable backup: A backup protected from modification or deletion for a defined period.

Review Questions

Short Answer Questions

1. What is the principal difference between object storage and block storage?
2. Define durability and explain how it differs from availability.
3. What is the purpose of a lifecycle policy?
4. State the meanings of RPO and RTO.
5. Why should a cache generally not be treated as the authoritative copy of important data?

Long Answer Questions

1. Compare object, block, and file storage, including interfaces, performance characteristics, and typical uses.
2. Explain how replication, consistency, backups, and versioning address different data-protection requirements.
3. Discuss the advantages and responsibilities associated with managed relational databases.
4. Explain how lifecycle tiers can reduce cost and identify the risks of poorly designed deletion policies.
5. Describe the ACID properties and compare relational databases with NoSQL services.

Analytical / Application Questions

1. An organisation stores video, customer orders, and shared design files. Recommend storage services for each data class and justify the choices using access patterns and recovery requirements.
2. A service requires an RPO of 15 minutes but currently performs a daily backup. Analyse the gap and propose technical and operational changes.
3. Design a data architecture for an online examination system using a managed database, object storage, and a cache. Explain consistency, security, backup, lifecycle, and failure-recovery decisions.

Multiple Choice Questions with Answers

Question 1

Which storage type is most appropriate for large, infrequently modified video files accessed through an API?

A. Object storage

B. Block storage

C. Local temporary disk

D. A database transaction log

Correct Answer: A. Object storage

Question 2

Which property describes the probability that data will not be lost over a specified period?

A. Latency

B. Availability

C. Durability

D. Throughput

Correct Answer: C. Durability

Question 3

What is the primary purpose of a cache-aside pattern?

A. To make the cache the only durable data store

B. To read from the cache first and populate it after a miss

C. To prevent all database writes

D. To replicate data across legal jurisdictions

Correct Answer: B. To read from the cache first and populate it after a miss

Question 4

Which ACID property ensures that all operations in a transaction succeed or none take effect?

A. Isolation

B. Durability

C. Consistency

D. Atomicity

Correct Answer: D. Atomicity

Question 5

An organisation requires no more than 15 minutes of data loss after a failure. Which requirement is being specified?

A. RTO

B. RPO

C. IOPS

D. SLA latency

Correct Answer: B. RPO

Question 6

Why is attaching one ordinary block volume to several independent hosts potentially dangerous?

A. The volume cannot store binary data

B. The hosts may corrupt shared file-system structures

C. Block storage always deletes old files

D. The volume automatically becomes object storage

Correct Answer: B. The hosts may corrupt shared file-system structures

Question 7

Which design is most suitable for high-volume key-value lookups with a known partition key?

A. A NoSQL key-value service

B. An archive storage tier

C. A file-sharing protocol alone

D. A monitoring dashboard

Correct Answer: A. A NoSQL key-value service

Question 8

If a lifecycle policy deletes objects after 30 days, which control is most important before production deployment?

A. Disable encryption

B. Remove versioning

C. Test retention, legal holds, and recovery requirements

D. Increase cache expiration indefinitely

Correct Answer: C. Test retention, legal holds, and recovery requirements

Question 9

A database replica has availability 0.99 and failures are assumed independent. What is the approximate availability of two replicas using A = 1 − (1 − a)^n?

A. 0.9900

B. 0.9950

C. 0.9999

D. 1.9900

Correct Answer: C. 0.9999

Question 10

Which statement best distinguishes a backup from replication?

A. Replication always costs less than backup

B. Backups can provide historical recovery, while replication may copy accidental changes quickly

C. Replication is only used for file storage

D. Backups cannot be encrypted

Correct Answer: B. Backups can provide historical recovery, while replication may copy accidental changes quickly