Cloud Storage and Database Systems

Introduction and Context

Artificial intelligence and machine learning systems are fundamentally data systems. Training datasets, feature tables, model checkpoints, embeddings, logs, labels, and real-time observations must be stored, retrieved, transformed, governed, and preserved. Cloud platforms provide many storage and database services, but these services are not interchangeable. A sound architecture selects each technology according to data structure, access pattern, consistency requirement, latency target, scalability, durability, cost, and governance obligations.

This chapter examines cloud storage models—object, block, and file storage—and database families including relational, NoSQL, vector, graph, and time-series systems. It also explains caching, data lakes, data warehouses, and lakehouses. The central design principle is that storage should be matched to workload rather than selected solely by popularity or provider branding.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish object, block, and file storage and select among them for AI workloads.

2. Explain consistency, availability, latency, scalability, and durability as architectural requirements.

3. Compare relational, NoSQL, vector, graph, and time-series databases.

4. Describe the roles of caches, data lakes, warehouses, and lakehouses.

5. Design a storage architecture for a data-intensive AI application.

6. Evaluate security, governance, cost, and operational limitations.

1. Storage Requirements in AI Systems

A storage decision begins with workload analysis. Important questions include:

• What is the data format: structured records, documents, images, audio, video, embeddings, or event streams?
• Is access sequential, random, transactional, analytical, or similarity-based?
• How much latency is acceptable?
• Must every read immediately observe the latest write?
• What volume and growth rate are expected?
• What retention, deletion, residency, and audit rules apply?

1.1 Core quality attributes

Latency is the time between a request and its response. Online recommendation or fraud detection may require millisecond-scale access, whereas model training can tolerate longer retrieval times if throughput is high.

Throughput is the volume of data transferred or operations completed per unit of time. Training pipelines often require high sequential-read throughput rather than extremely low single-request latency.

Consistency describes how read operations relate to completed writes. Strong consistency gives a read the latest committed value according to the service's contract. Eventual consistency permits temporary differences between replicas but can provide useful availability and scalability. Consistency must be specified precisely; a provider's implementation and API contract determine the actual guarantee.

Availability is the proportion of time a service is usable. Durability is the likelihood that stored data will not be permanently lost. These are different: a service may be temporarily unavailable while preserving data, or available while a poorly configured object is accidentally deleted.

Scalability is the ability to accommodate increasing data, users, or operations. Vertical scaling increases the capacity of an individual resource; horizontal scaling distributes work across resources.

A useful way to reason about architecture is to separate the system of record from derived stores. For example, raw images may be preserved in object storage, metadata may be held in a relational database, and embeddings may be indexed in a vector database. The derived stores can be rebuilt from authoritative data when appropriate.

2. Cloud Storage Models

2.1 Object storage

Object storage stores data as objects within logical containers commonly called buckets. An object normally consists of content, a key or name, system metadata, and optional user metadata. Access is generally through APIs rather than traditional operating-system file operations.

Object storage is well suited to:

• Images, audio, video, documents, and training corpora.
• Data lakes and model checkpoints.
• Backups, archives, and immutable evidence.
• Large sequential files and event exports.

Its strengths include very high capacity, strong durability designs, lifecycle policies, versioning, replication options, and relatively low cost. It is not normally ideal for workloads requiring frequent in-place updates to small regions of a large file or POSIX-style file locking. Applications typically replace an object rather than modify a byte range through ordinary file semantics.

Object keys resemble paths, but this does not necessarily mean that the underlying system is a hierarchical file system. Listing and prefix operations may have different performance and cost characteristics from local directory operations. Applications should avoid assuming that renaming an object is always an inexpensive metadata operation.

For AI, object storage is commonly the durable foundation for raw data, curated data, feature files, experiment artifacts, and model versions. Metadata should identify source, consent status, schema version, preprocessing code, and lineage.

2.2 Block storage

Block storage presents a volume of addressable blocks to a virtual machine or host. The operating system creates a file system or database format on the volume. This model supports random reads and writes, low-latency access, and familiar disk semantics.

Block storage is appropriate for:

• Database data and transaction logs.
• Virtual-machine boot volumes.
• Scratch space for preprocessing.
• High-performance local indexes and temporary training data.

Block volumes are usually attached to particular compute resources or availability zones, although some platforms provide specialized shared or replicated variants. They require management of file systems, partitioning, encryption, snapshots, and capacity. A block volume is not automatically a complete backup strategy: snapshots must be retained, tested, and protected from accidental deletion.

2.3 File storage

File storage exposes a shared hierarchical namespace using protocols such as NFS or SMB. Multiple compute instances can access common files, subject to permissions and locking semantics.

File storage is useful when applications expect:

• Directories and file paths.
• Shared project files.
• Home directories and content-management workflows.
• Legacy applications that cannot use object APIs.

It can simplify shared access but may be more expensive or less scalable than object storage for massive data collections. Performance depends on metadata operations, directory structure, network conditions, and the selected service tier. File and object interfaces should not be treated as identical: they differ in naming, locking, update semantics, and consistency behavior.

2.4 Comparison

Object storage emphasizes scalable durable objects and API access. Block storage emphasizes low-latency virtual disks. File storage emphasizes shared hierarchical access. A typical ML platform may use block storage for a database's active working set, file storage for shared preprocessing tools, and object storage for durable datasets and model artifacts.

3. Database Families

Storage preserves bytes; databases additionally organize data, enforce constraints, provide query languages or APIs, and manage concurrent operations.

3.1 Relational databases

Relational databases represent data in tables of rows and columns. A schema defines types and relationships. SQL supports filtering, joining, aggregation, and transactions. Primary keys identify records, while foreign keys express relationships between tables.

Relational systems are effective when correctness, structured data, and transactional integrity matter. Examples include order systems, user accounts, billing, inventory, and metadata catalogs. A transaction is commonly discussed using ACID properties:

• Atomicity: a transaction's operations succeed together or fail together.
• Consistency: committed transactions preserve defined constraints.
• Isolation: concurrent transactions are controlled so that intermediate states are not improperly observed.
• Durability: committed data survives specified failures.

Normalization can reduce duplication and update anomalies, while carefully chosen denormalization can improve read performance. Indexes accelerate selected queries but consume storage and impose write overhead. Relational databases are not automatically unsuitable for AI; they are often the authoritative source for labels, users, experiments, and feature metadata.

3.2 NoSQL databases

“NoSQL” describes several non-relational approaches rather than one uniform technology.

Key-value stores map a key to a value and are useful for sessions, counters, configuration, and fast lookups. They normally require access patterns to be designed in advance.

Document databases store semi-structured documents, often represented as JSON-like objects. They support evolving records and nested data, but cross-document transactions and joins may be less central than in relational systems.

Wide-column databases organize data by partition and clustering keys. They can provide high write and read scalability for carefully designed access patterns.

NoSQL systems often trade general-purpose relational querying for horizontal scalability, flexible schemas, or predictable access paths. They do not eliminate data modeling; they make workload-driven modeling especially important. Poor partition-key selection can create hotspots, uneven load, or unbounded partitions.

3.3 Vector databases

A vector database stores numerical representations, or embeddings, and supports nearest-neighbor search. An embedding maps an item such as text, an image, or a product into a numerical vector such that useful relationships may be reflected by geometric proximity.

A similarity measure must be selected consistently with the embedding model. For vectors x and y, cosine similarity is commonly defined as:

cos(x,y) = (x · y) / (||x|| ||y||),

where x · y is the dot product and ||x|| is the Euclidean norm. The formula applies to nonzero vectors. If vectors are normalized to unit length, cosine similarity equals their dot product. A distance or similarity measure is meaningful only when the embedding model and indexing method support it.

Vector search may be exact or approximate. Approximate nearest-neighbor indexes reduce search time at the possible cost of missing some true nearest neighbors. Important evaluation measures include retrieval latency, recall at k, memory use, update cost, and filtering behavior. A production retrieval-augmented generation system often combines vector similarity with metadata filters, keyword search, access-control checks, and a reranker. A vector index should usually be treated as a derived representation, not the sole authoritative copy of source documents.

3.4 Graph databases

Graph databases represent entities as nodes and relationships as edges, often with properties on both. They are valuable when the structure and traversal of relationships are central to the query.

Applications include fraud-ring detection, identity resolution, recommendation, knowledge graphs, supply-chain analysis, and dependency mapping. A graph can express multi-hop questions naturally, such as finding accounts connected through shared devices or organizations. However, graph systems may not be the best choice for simple aggregate reporting or very large scans of independent records. The graph model should correspond to the questions the application must answer.

3.5 Time-series databases

Time-series databases are optimized for measurements indexed by time and often by dimensions such as device, region, or service. They commonly provide efficient append operations, time-window queries, downsampling, retention policies, and aggregations.

They are appropriate for telemetry, sensor readings, infrastructure metrics, financial observations, and model-monitoring signals. Design concerns include timestamp accuracy, out-of-order events, cardinality of tags, retention, and aggregation resolution. AI monitoring may store prediction scores, drift indicators, latency, and resource metrics in a time-series system while preserving detailed events in object storage.

4. Caching and Data Movement

A cache stores copies of data closer to consumers to reduce latency and load on an origin system. Common cacheable items include feature values, popular documents, authorization results, database query results, and model responses.

A cache introduces a second copy and therefore a consistency problem. Common strategies include:

• Cache-aside: the application reads the cache, retrieves from the source on a miss, and then populates the cache.
• Write-through: writes update the cache and underlying store through a coordinated path.
• Write-back: writes may be acknowledged by the cache before later persistence, increasing performance but also risk.
• Time-to-live expiration: entries become invalid after a configured interval.
• Explicit invalidation: an event removes or updates affected entries.

Cache-aside is flexible but can return stale data and may suffer a “cache stampede” when many requests miss simultaneously. Mitigations include request coalescing, jittered expiration, rate limiting, and fallback behavior. Sensitive data requires encryption, access controls, and careful consideration of whether cached copies violate retention or deletion requirements.

5. Data Lakes, Warehouses, and Lakehouses

5.1 Data lakes

A data lake stores large quantities of raw or lightly processed data, often in object storage. It can hold structured, semi-structured, and unstructured data. Its flexibility supports exploratory analytics, data science, and reuse of original records.

A lake is not simply a bucket filled without organization. Effective lakes require cataloging, schemas, partitioning, quality checks, ownership, lineage, retention, and access control. “Schema-on-read” allows interpretation at query time, but uncontrolled schema variation can create confusion and unreliable analyses.

5.2 Data warehouses

A data warehouse is optimized for governed analytical queries over structured, integrated data. It commonly uses column-oriented storage, compression, parallel execution, and dimensional or relational models. Warehouses support business intelligence, reporting, historical analysis, and curated feature computation.

A warehouse usually prioritizes consistent definitions and predictable analytical performance over unrestricted raw-data flexibility. Loading and transformation processes must address late-arriving data, slowly changing dimensions, duplicates, and data-quality failures.

5.3 Lakehouses

A lakehouse combines low-cost, flexible object storage with warehouse-like table management, transactions, schema enforcement, and analytical performance. Table formats and processing engines may provide atomic commits, version history, partition management, and support for batch and streaming workloads.

The term describes an architectural pattern rather than one universal product. A lakehouse can reduce duplication between a data lake and warehouse, but it does not automatically solve governance, query optimization, security, or semantic consistency. Teams must still define data products, quality expectations, and ownership.

5.4 AI data architecture

A practical AI data platform often contains several layers:

1. Ingestion: events, files, databases, sensors, and external sources.

2. Raw zone: immutable or versioned source data in object storage.

3. Curated zone: validated, standardized, and documented datasets.

4. Feature and label stores: reusable training and serving features with point-in-time correctness.

5. Training and evaluation stores: reproducible snapshots, experiment metadata, and model artifacts.

6. Serving databases and caches: low-latency access to predictions, features, documents, or embeddings.

7. Monitoring stores: operational metrics, traces, feedback, and drift observations.

Point-in-time correctness is particularly important in supervised learning. A training feature must reflect only information that would have been available at the prediction time; otherwise, data leakage can produce deceptively high evaluation results.

6. A Selection Framework

The following sequence supports technology selection.

Step 1: Define the access pattern. Identify reads, writes, scans, joins, traversals, similarity searches, and time-window queries.

Step 2: Define correctness requirements. Determine whether stale reads are acceptable, which operations require transactions, and how conflicts are resolved.

Step 3: Define scale and performance. Estimate data volume, growth, concurrency, latency percentiles, throughput, and peak behavior. Average latency alone can hide unacceptable tail latency.

Step 4: Separate authoritative and derived data. Preserve recoverable source data and document how indexes, features, and aggregates can be rebuilt.

Step 5: Evaluate failure and recovery. Specify backup frequency, recovery point objective (maximum tolerable data loss measured in time), and recovery time objective (maximum tolerable restoration time). Test restoration rather than merely enabling backups.

Step 6: Evaluate cost and operations. Include storage, requests, data transfer, compute, replicas, backups, indexing, personnel, and migration costs.

Step 7: Validate with a representative workload. Benchmark realistic data distributions, concurrency, failures, updates, and security controls. A synthetic benchmark that omits skew or cold-start behavior may mislead.

A compact decision example is as follows: raw medical images should generally use governed object storage; patient and study metadata may require a relational database; image embeddings may use a vector index; device measurements may use a time-series database; and frequently accessed authorization data may use a cache. The exact design depends on regulations, query patterns, and provider capabilities.

7. Illustrative Case: Cloud Platform for Medical-Imaging Assistance

Background

The following is a fictional illustrative case. A healthcare organization develops an AI system that assists clinicians by finding similar historical images and displaying a preliminary classification. The system receives images from hospitals, stores annotations, trains models periodically, and serves results during clinical workflows.

Problem

The organization must preserve original images, support rapid clinical queries, prevent unauthorized access, maintain reproducible training datasets, and comply with retention and audit obligations. It also needs to handle bursts when several hospitals upload studies simultaneously.

Analysis

Original images are large binary objects and should not be placed directly in a transactional metadata table. Clinical metadata—study identifier, patient pseudonym, acquisition time, modality, annotation status, and consent information—requires structured constraints and carefully controlled transactions. Similarity search requires embeddings, but embeddings alone cannot provide a complete clinical audit trail. Training requires versioned datasets and protection against leakage between related studies.

Approach and Solution

The design uses:

• Encrypted, versioned object storage for original images, derived images, labels, and model artifacts.
• A relational database for authorized metadata, workflow state, annotation records, and audit references.
• A vector database for image embeddings, with metadata filters enforcing modality and authorization constraints.
• A cache for short-lived, non-sensitive results and frequently accessed configuration, with explicit invalidation when model versions change.
• A lakehouse-style curated layer for reproducible analytical and training tables.
• A time-series store for service latency, error rates, resource measurements, and model-monitoring signals.

A catalog records dataset versions, preprocessing code, model version, source lineage, and retention classification. Access is granted through least privilege, with encryption in transit and at rest. Object deletion and patient-rights processes are designed to propagate to derived indexes and caches where legally required.

Outcome

The architecture separates durable source data from specialized serving systems. Clinicians receive low-latency similarity results, while researchers can reproduce a training run from versioned inputs. Operational teams can monitor latency and drift independently of clinical records.

Learning Points

First, no single database satisfies all access patterns efficiently. Second, a vector index is a derived service and must be governed like other copies of sensitive data. Third, performance, reproducibility, and privacy must be designed together. Finally, backups and deletion workflows must include indexes, caches, and exported datasets—not only the primary database.

8. Security, Governance, and Ethical Issues

Cloud storage expands the number of copies and interfaces through which data can be accessed. Essential controls include identity-based access, least privilege, encryption, network segmentation, key management, audit logging, vulnerability management, and tested backup restoration.

Data classification should distinguish public, internal, confidential, and regulated information. Personally identifiable information and protected health or financial information require appropriate legal and organizational safeguards. Data minimization reduces risk: retain what is needed for a defined purpose and avoid copying sensitive fields into logs or caches.

AI introduces additional concerns. Training data may contain hidden demographic bias, copyrighted material, or information collected without meaningful consent. Dataset lineage and documentation support accountability. Access-control filters must be applied before retrieval results reach a model or user. Deletion requests are complicated when data has been transformed into embeddings, aggregates, backups, or trained models; organizations need documented policies for propagation and model retraining where required.

Cross-region replication and data transfer may create residency or sovereignty issues. Contracts with cloud providers should clarify responsibilities, incident notification, subcontractors, deletion procedures, and service-level commitments. Governance is not a substitute for technical controls, and technical controls cannot replace lawful purpose and responsible stewardship.

9. Contemporary Developments and Limitations

Modern platforms increasingly support unified batch and streaming tables, serverless query engines, multimodal vector retrieval, managed feature stores, and automated data-quality checks. Open table formats and interoperable catalogs can reduce dependence on a single processing engine. Specialized hardware and storage tiers are improving large-scale training throughput.

Nevertheless, managed services can create vendor lock-in through proprietary APIs, indexes, formats, and operational practices. Migration may be difficult when data is large or egress charges are significant. Abstracting every provider feature may reduce performance, while relying entirely on proprietary features reduces portability. Organizations should identify exit strategies, maintain exportable representations where practical, and test restore and migration procedures.

Storage technology also has limits. A database cannot compensate for poorly defined data semantics. A cache cannot make an inherently slow or incorrect source reliable. A vector search result is not automatically relevant, fair, or safe. Benchmarks must include realistic distributions, authorization checks, failures, and total cost.

Best Practices

1. Start with workload and data-classification requirements rather than product names.

2. Keep authoritative data versioned and make derived stores reproducible.

3. Document consistency, retention, recovery, and deletion semantics.

4. Partition and index according to actual access patterns; monitor skew and hotspots.

5. Use caches deliberately and measure staleness as well as hit rate.

6. Test backups, disaster recovery, access controls, and migration paths.

7. Apply point-in-time correctness to training features and labels.

8. Monitor latency percentiles, throughput, errors, storage growth, and cost.

9. Treat embeddings, logs, exports, and backups as governed data copies.

10. Validate AI retrieval quality, privacy, bias, and authorization—not only infrastructure performance.

Chapter Summary

Cloud storage and database systems provide different abstractions and guarantees. Object storage is suited to durable, large-scale files and AI artifacts; block storage provides low-latency virtual disks; and file storage provides shared hierarchical access. Relational databases support structured data, constraints, and transactions, while NoSQL systems emphasize workload-specific scalability and flexible models. Vector databases support embedding similarity, graph databases support relationship traversal, and time-series databases support time-indexed observations.

Caches reduce latency but introduce staleness and invalidation challenges. Data lakes preserve flexible raw data, warehouses provide governed analytical processing, and lakehouses combine aspects of both. AI platforms commonly use several of these technologies together, separating authoritative data from derived indexes, features, aggregates, and caches. Selection should consider access patterns, consistency, latency, scale, durability, recovery, cost, security, and governance. Responsible architecture also addresses lineage, privacy, deletion, bias, regulatory obligations, and reproducibility.

Key Terms

Object storage: API-based storage of data objects and metadata.

Block storage: Addressable virtual disk storage used by operating systems and databases.

File storage: Shared hierarchical storage accessed through file protocols.

Consistency: The relationship between reads and committed writes.

Durability: Protection against permanent data loss.

Availability: The degree to which a service is operational and accessible.

Relational database: A database organized into related tables and queried commonly with SQL.

NoSQL database: A broad class of non-relational databases designed for particular scalability or data-model needs.

ACID: Atomicity, consistency, isolation, and durability properties of transactions.

Vector database: A database optimized for storing embeddings and performing similarity search.

Embedding: A numerical representation of an item used for machine-learning comparison or retrieval.

Graph database: A database that models entities and relationships as nodes and edges.

Time-series database: A database optimized for timestamped measurements and time-window queries.

Cache: A faster temporary copy of data used to reduce latency or origin load.

Data lake: Scalable storage for raw or lightly processed data of varied types.

Data warehouse: A governed analytical system optimized for structured queries.

Lakehouse: An architecture combining flexible lake storage with warehouse-like table management.

Data lineage: Information describing data origin, transformations, and dependencies.

Point-in-time correctness: Ensuring training data contains only information available at the relevant historical time.

Recovery point objective: The maximum acceptable amount of data loss expressed as time.

Review Questions

Short Answer Questions

1. What are the principal differences between object, block, and file storage?

2. Why is a vector database usually considered a derived store in an AI architecture?

3. Define strong consistency and eventual consistency.

4. What problem does a cache solve, and what new problem can it introduce?

5. State two characteristics of a time-series database.

Long Answer Questions

1. Compare relational and NoSQL databases with respect to schema, transactions, scalability, and workload design.

2. Explain how a lake, warehouse, and lakehouse differ and identify suitable uses for each.

3. Discuss the role of object storage in an end-to-end machine-learning pipeline.

4. Explain ACID properties and their importance in an AI metadata or labeling system.

5. Describe security and governance requirements for cloud systems containing sensitive AI data.

Analytical / Application Questions

1. Design a storage architecture for an online recommendation service that uses product images, user transactions, embeddings, and real-time events. Justify each technology.

2. A team stores raw data in object storage, features in a warehouse, and embeddings in a vector database. Explain how it could maintain lineage, reproducibility, authorization, and deletion across these systems.

3. A similarity-search service has high average performance but unacceptable latency during traffic bursts. Analyze possible causes and propose changes involving caching, partitioning, indexing, capacity, and workload testing.

Multiple Choice Questions with Answers

Question 1

Which storage model is generally most appropriate for durable collections of images and model artifacts?

A. Object storage

B. Block storage

C. CPU register storage

D. Relational table pages

Correct Answer: A. Object storage

Question 2

Which database family is primarily designed for nearest-neighbor search over embeddings?

A. Time-series database

B. Vector database

C. Graph database

D. Relational database

Correct Answer: B. Vector database

Question 3

What is a principal risk introduced by caching data?

A. All data becomes permanently durable

B. The cache eliminates the need for authorization

C. Readers may observe stale data

D. Transactions automatically become serializable

Correct Answer: C. Readers may observe stale data

Question 4

Which ACID property means that a transaction's operations succeed together or fail together?

A. Isolation

B. Durability

C. Consistency

D. Atomicity

Correct Answer: D. Atomicity

Question 5

Which system is especially suitable for traversing multi-hop relationships such as fraud rings?

A. Graph database

B. Object store

C. Time-series database

D. Block volume

Correct Answer: A. Graph database

Question 6

Why is point-in-time correctness important in machine-learning training data?

A. It increases object-storage capacity

B. It prevents use of information that was unavailable at prediction time

C. It guarantees every model is unbiased

D. It removes the need for labels

Correct Answer: B. It prevents use of information that was unavailable at prediction time

Question 7

Which architecture combines flexible object-based data storage with warehouse-like table management and transactions?

A. Cache-only architecture

B. File server

C. Lakehouse

D. Key-value store

Correct Answer: C. Lakehouse

Question 8

A block-storage volume is most directly analogous to which resource from an operating system's perspective?

A. A virtual disk

B. A graph edge

C. A search query

D. A message queue

Correct Answer: A. A virtual disk

Question 9

For nonzero vectors x and y, cosine similarity is computed as:

A. ||x|| + ||y||

B. (x · y)/(||x|| ||y||)

C. x · y + ||x||

D. ||x − y||/(x · y)

Correct Answer: B. (x · y)/(||x|| ||y||)

Question 10

Which database is optimized for timestamped measurements and time-window aggregation?

A. Document database

B. Graph database

C. Time-series database

D. Vector database

Correct Answer: C. Time-series database