Cloud Storage, Databases and Data Architecture

Introduction

Cloud data architecture is the disciplined design of storage, databases, data movement, processing, security and governance across distributed computing environments. It is not simply a choice between a cloud provider's storage products. A sound architecture must match the characteristics of data and workloads with requirements for performance, durability, availability, consistency, cost, sovereignty, retention and regulatory control.

A transactional banking workload, an image archive, a container's boot disk, a recommendation engine and a data-warehouse report have different access patterns. Treating them as the same problem creates unnecessary cost and operational risk. This chapter develops a decision framework for selecting object, block and file storage; relational, NoSQL, graph, cache and analytical databases; and for designing data lakes, lakehouses and reliable pipelines.

Learning objectives

By the end of this chapter, a learner should be able to:

1. Distinguish object, block and file storage and select among them for realistic workloads.
2. Compare relational, NoSQL, graph, cache and analytical database models.
3. Explain partitioning, replication, consistency, durability and availability trade-offs.
4. Design a governed data lake, lakehouse and batch or streaming pipeline.
5. Apply lifecycle, sovereignty, security and compliance requirements to architecture decisions.
6. Evaluate a proposed architecture using workload, operational and business criteria.

1. Data architecture as a set of design decisions

A useful architecture begins with workload analysis rather than product names. Important questions include:

• What is the data's structure: tabular, document, key-value, graph, media or event data?
• Is access primarily sequential, random, transactional, analytical or streaming?
• What are the read/write rates, object sizes, latency targets and growth rate?
• Is the data authoritative, derived, temporary, cached or archival?
• Which failures must the design tolerate, and how quickly must service recover?
• Where may data be stored, processed and replicated?
• What retention, deletion, legal hold and audit requirements apply?

Two terms must be separated. Durability is the probability that stored data will not be lost over a defined period. Availability is the probability that a service is accessible when requested. Replication can improve both, but it does not make them identical: a highly available system can continue serving stale or corrupted data, while a durable archive may be temporarily inaccessible.

For a simple storage-cost estimate, if data volume is V gigabytes, the monthly price is p currency units per gigabyte-month, and an additional monthly request, transfer and replication cost is C_o, then:

This is an estimate, not a universal billing formula. Providers may charge by binary units, storage class, operations, retrieval, minimum object duration, network path and replication. A professional estimate therefore models each relevant billing component separately.

2. Storage services

2.1 Object storage

Object storage stores an object consisting of content, metadata and a unique key within a bucket or similar namespace. Objects are accessed through APIs, usually over HTTP, rather than by mounting a traditional disk. Object storage is designed for very large scale and commonly provides high durability, lifecycle policies, versioning, encryption and policy-based access control.

Typical uses include:

• documents, photographs, video and backups;
• data-lake raw and curated zones;
• software packages and static web assets;
• logs, exports and machine-learning datasets.

Object storage is generally a poor choice for a database's low-latency random-write data files unless the database is specifically designed for object storage. Updates usually replace an object or create a new version rather than modifying arbitrary bytes in place. Listing and rename semantics may also differ from those of a local file system. Multipart upload, checksums and versioning are important for large or valuable objects.

Object storage classes commonly represent a cost-versus-access trade-off: frequent-access, infrequent-access and archival classes. Lifecycle rules can transition objects after an age threshold or delete them after retention expires. A rule must account for minimum storage durations, retrieval charges, legal holds and the possibility that compliance requires immutable retention.

2.2 Block storage

Block storage presents a virtual block device to a virtual machine or specialized compute service. The operating system formats it with a file system, and applications see sectors or blocks rather than objects. It is suitable for boot volumes, transactional databases, virtual-machine file systems and applications requiring low-latency random I/O.

Performance is influenced by IOPS (input/output operations per second), throughput and latency. These are different properties. A workload using small random reads may be IOPS-limited; a workload scanning large files may be throughput-limited. Provisioned performance must be tested against the application's block size, queue depth and concurrency rather than selected from a single headline number.

Snapshots are point-in-time representations of volumes, often implemented efficiently through changed-block tracking. They are useful for recovery but are not automatically a complete backup strategy. A recovery design must test restoration, preserve required application consistency and protect snapshots from accidental deletion or ransomware.

2.3 File storage

File storage presents a hierarchical namespace of directories and files, commonly through protocols such as NFS or SMB. It is useful when applications require shared access, familiar file semantics, directory permissions or concurrent mounts. Examples include shared content management, engineering files, home directories and legacy applications.

File systems introduce coordination concerns: locks, concurrent updates, metadata hot spots, directory scaling and network latency. A shared file service should not be selected merely because an application uses files; object storage may be more scalable for immutable content, while block storage may be preferable for a single host's database.

2.4 Storage selection

A practical decision process is: identify access semantics; quantify latency, IOPS and throughput; classify data value and retention; identify sharing and concurrency needs; then compare cost and failure behavior. Storage abstraction should be deliberate, not accidental.

3. Database models and workload fit

3.1 Relational databases

Relational databases represent data in tables with defined schemas and relationships. SQL supports declarative querying, joins, constraints and transactions. The relational model is strong where data integrity and multi-record transactions are central—for example, orders, payments, inventory and student registration.

ACID describes important transaction properties: atomicity, consistency, isolation and durability. ACID does not guarantee that every distributed deployment has zero latency or unlimited scale. Designers must still choose transaction boundaries, indexes, isolation levels, failover behavior and backup strategies.

Normalization reduces update anomalies by separating facts into related tables. Denormalization may be justified for read performance, reporting or distributed access, but duplicated data then requires explicit synchronization and ownership rules.

3.2 NoSQL databases

“NoSQL” covers several models rather than one alternative model:

• Key-value stores retrieve a value by key and are effective for sessions, profiles and simple lookups.
• Document databases store JSON-like documents and suit aggregates whose fields evolve together.
• Wide-column databases organize data by partition and clustering keys and suit very large, distributed workloads.
• Time-series databases optimize timestamped measurements and retention-oriented queries.

NoSQL design is commonly query-driven: first identify access patterns, then choose partition keys and data layout. A poorly chosen partition key can create a hot partition, uneven storage or unbounded scans. Flexible schema does not eliminate governance; it makes validation, versioning and data contracts more important.

3.3 Graph databases

A graph database models entities as vertices and relationships as edges, optionally with properties. It is appropriate when relationship traversal is central: fraud networks, identity relationships, supply chains, knowledge graphs and recommendation paths. A relational schema can represent graphs, but repeated multi-hop traversal may be more natural in a graph engine. Graph systems are not automatically superior for aggregate reporting or simple tabular transactions.

3.4 Caches

A cache stores data closer to the consumer to reduce latency and load on an authoritative system. It is not normally the system of record. Important policies include time-to-live (TTL), eviction, invalidation, negative caching and behavior during cache failure.

The cache hit ratio is:

where H is the number of hits and M is the number of misses during the same observation interval. The formula assumes hits and misses are counted consistently. A high hit ratio does not prove correctness: stale or incorrectly invalidated values can still produce wrong results.

Common patterns include cache-aside, in which the application reads the cache and loads a miss from the database, and write-through, in which writes update the cache and backing store through a coordinated mechanism. Cache-aside is flexible but requires careful invalidation and protection against a thundering herd of simultaneous misses.

3.5 Analytical databases

Analytical databases optimize scans, aggregation and reporting over large datasets. Columnar storage reads only required columns and compresses similar values effectively. Analytical systems may separate compute from storage and scale elastic query resources. They are not necessarily appropriate for high-frequency, single-row transactional updates.

Operational databases and analytical stores are often separated through change data capture, event streams or scheduled extraction. The design must address lag, duplicate events, schema evolution, late-arriving data and reconciliation with the source of truth.

4. Distribution, consistency and resilience

4.1 Partitioning and sharding

Partitioning divides data into subsets. Horizontal partitioning (sharding) distributes rows or documents; vertical partitioning separates columns or attributes. Good partitioning balances storage and traffic while supporting common queries. A partition key should have sufficient cardinality, avoid predictable hot spots and align with query predicates.

A design should specify what happens when a partition becomes unavailable, when a node is added, and when data must be rebalanced. Rebalancing creates network and I/O load and can affect latency.

4.2 Replication

Replication maintains multiple copies or replicas. Synchronous replication confirms a write after required replicas acknowledge it, usually improving consistency at the cost of write latency and reduced tolerance for network separation. Asynchronous replication acknowledges earlier but introduces replication lag and possible data loss after a primary failure.

For a simple independent-failure illustration, if each of n replicas has probability p of being unavailable during an interval and failures are assumed independent, the probability that all replicas are unavailable is:

This is only a simplified model. Real failures are correlated by zones, power, software defects, credentials and operator actions. Therefore, replicas should be distributed across meaningful failure domains and the architecture should be evaluated against correlated failures.

4.3 Consistency models

Strong consistency makes a successful read reflect the most recent completed write according to the system's defined ordering. Eventual consistency permits temporary divergence while replicas converge. Session guarantees such as read-your-writes can provide useful application behavior without global synchronous coordination.

Consistency is a workload decision. A payment authorization normally needs strong integrity controls; a social-media view counter may tolerate delayed convergence. The design must document conflict resolution, ordering, duplicate handling and what users observe during network partitions.

Recovery objectives should be explicit:

• RPO (Recovery Point Objective): maximum tolerable amount of data loss measured in time.
• RTO (Recovery Time Objective): maximum tolerable time to restore service.

A backup that exists but has never been restored is an unverified assumption. Recovery exercises should test credentials, dependencies, data integrity, application replay and communication procedures.

5. Data lakes, lakehouses and pipelines

5.1 Data lake

A data lake is a scalable repository, commonly based on object storage, that retains raw and processed data in many formats. A typical organization uses zones such as raw, validated, curated and consumption. Raw data should preserve provenance and original meaning; curated data should be easier and safer to consume.

A lake can become a “data swamp” when ownership, metadata, quality, access control and retention are absent. File formats, partitioning, naming, cataloguing and data contracts are architectural concerns, not clerical details. Columnar formats such as Parquet are often effective for analytical scans, while excessive small files can damage query performance.

5.2 Lakehouse

A lakehouse combines low-cost, flexible lake storage with database-like table management, including schema enforcement or evolution, transactional updates, version history and governance features. It is useful when the organization wants one governed data foundation for batch analytics, streaming, business intelligence and machine learning.

A lakehouse does not remove the need for modeling. Teams still need ownership, quality rules, partition strategy, workload isolation, access policies and cost controls. Table transactions also have scope and concurrency limits that must be checked for the selected implementation.

5.3 Pipeline architecture

A pipeline moves and transforms data from producers to consumers. A robust pipeline commonly includes ingestion, validation, quarantine, transformation, storage, cataloguing, serving and monitoring.

Batch pipelines process bounded data on a schedule. Streaming pipelines process events continuously and must address ordering, replay, windows, back pressure and late arrival. “Exactly once” is often a system-level goal rather than a simple property: durable offsets, idempotent writes and deduplication are required across boundaries.

An idempotent operation produces the same intended result when safely repeated. For example, writing an order using a stable order identifier with an upsert rule is safer than blindly inserting every retried message. Event schemas should be versioned, and producers should publish compatibility expectations.

A pipeline's quality controls should include completeness, validity, uniqueness, timeliness and consistency. Observability should expose ingestion lag, failed records, throughput, schema changes, cost and lineage. A quarantine path prevents malformed data from silently contaminating trusted datasets.

Figure: A governed data flow from source systems through ingestion, storage and consumption.

6. Governance, sovereignty and security

Data governance assigns decision rights and controls across the data lifecycle. It includes ownership, classification, metadata, quality, access, retention, lineage and accountability. A data catalogue should identify business meaning, owner, sensitivity, source, freshness and permitted use—not merely file locations.

Data sovereignty concerns the legal and policy authority associated with data location, processing and control. Residency requirements may restrict storage to a country or region; sectoral rules may restrict transfers or require approved subprocessors. Replicating data for availability can unintentionally create an unauthorized copy. Architects should map data flows, regions, administrative access and backup destinations, then obtain legal and compliance review where required.

Security controls include encryption in transit and at rest, customer-managed keys where justified, least-privilege identities, network segmentation, private endpoints, secrets management, immutable backups and tamper-evident audit logs. Encryption does not solve excessive permissions or insecure application behavior. Tokenization, masking and pseudonymization can reduce exposure, but pseudonymized data may remain personal data under applicable law.

Retention is a controlled lifecycle, not indefinite accumulation. Policies should distinguish operational retention, archival retention, legal hold and secure deletion. Deletion must cover indexes, replicas, caches, snapshots and derived copies where required. AI and machine-learning systems add concerns about training-data consent, provenance, memorization, bias and the right to correct or delete data.

7. Illustrative case: a regional health platform

Background

A fictional regional health platform receives appointment records, clinician notes, diagnostic images, device measurements and billing transactions. It serves hospitals in two jurisdictions with different residency rules. Clinicians need reliable transactions and fast patient lookup; analysts need longitudinal research datasets.

Problem

The existing system stores all data in one relational database. Image files increase storage cost, reports slow operational queries, and a nightly export creates a twelve-hour analytical delay. The organization also lacks a tested disaster-recovery process and cannot prove where backups reside.

Analysis

Patient registration, appointments and billing require relational transactions and constraints. Images are large immutable objects and need metadata linking them to patients and studies. Device measurements are high-volume time-series data. Research queries require a governed analytical environment, but direct analyst access to identifiable operational tables would create privacy and performance risks.

Approach

The proposed architecture uses:

1. A relational database for authoritative patient, appointment and billing records, with zone-aware replicas and tested point-in-time recovery.
2. Object storage for encrypted diagnostic images, with separate keys and lifecycle rules. Metadata remains linked through controlled identifiers rather than filenames containing sensitive information.
3. A time-series or distributed NoSQL store for device readings, partitioned by facility and time while avoiding a single hot facility partition.
4. A change-data-capture stream for approved operational changes, with schema validation, durable offsets, replay and idempotent loading.
5. A lakehouse with raw, restricted and curated zones. Identifiers are tokenized before research access; row-, column- and purpose-based policies restrict use.
6. A catalogue recording owner, classification, lineage, freshness and residency. Replicas and backups are constrained to approved jurisdictions.
7. A recovery exercise that restores databases, rehydrates services, validates image references and measures RPO and RTO.

Outcome

The design separates transactional and analytical workloads, reduces unnecessary database growth and provides fresher analytical data. Its success is not measured only by query speed: evidence of correct access control, tested recovery, complete lineage and lawful deletion is equally important.

Learning points

The case demonstrates polyglot persistence without indiscriminate technology multiplication. Each store has a defined system-of-record or derived role. It also shows that sovereignty, lifecycle and recovery must be designed with replication and pipelines rather than added after deployment.

8. Best practices and limitations

Best practice is to define data products and ownership, document access patterns, select storage by semantics, separate systems of record from caches and derived stores, and automate policy enforcement. Use infrastructure as code, versioned schemas, contract tests, encryption, private connectivity, least privilege and cost tagging. Monitor storage growth, query performance, cache behavior, replication lag, data quality and recovery readiness.

Avoid assuming that one database solves every workload. Avoid choosing a partition key without realistic traffic tests, treating snapshots as backups, using eventual consistency without documenting user-visible effects, or retaining raw data forever. Managed services reduce undifferentiated operations but do not eliminate vendor lock-in, quota limits, outages, migration complexity or responsibility for configuration and data governance.

Contemporary architectures increasingly use open table formats, metadata-driven pipelines, data contracts, streaming ingestion, confidential computing and policy-as-code. AI assistants can help classify schemas, generate mappings or detect anomalies, but generated transformations require review, testing and provenance. Sensitive data should not be exposed to an AI service without an approved processing basis and appropriate controls.

Chapter Summary

Cloud data architecture matches data semantics and workload behavior to storage and database capabilities. Object storage provides scalable API-oriented storage for content, backups and lake data; block storage provides virtual disks for low-latency random I/O; and file storage provides shared hierarchical access. Relational systems emphasize integrity and transactions, NoSQL systems serve specialized distributed access patterns, graph systems emphasize relationships, caches reduce latency, and analytical databases optimize scans and aggregation.

Partitioning, replication and consistency determine how a system behaves under scale and failure. RPO and RTO make recovery requirements measurable. Data lakes preserve broad, often raw datasets, while lakehouses add governed table management and transactional behavior. Reliable pipelines require validation, schema evolution, idempotency, replay, observability and lineage.

Finally, architecture is incomplete without governance. Classification, sovereignty, residency, encryption, least privilege, retention, deletion, auditability and tested recovery must span primary data and every replica, cache, snapshot and derived dataset.

Key Terms

Object storage: API-oriented storage of content, metadata and a key.

Block storage: Virtual disk storage presented as addressable blocks.

File storage: Hierarchical shared storage accessed through file protocols.

Durability: Resistance of stored data to loss over a defined period.

Availability: Likelihood that a service is accessible when requested.

IOPS: Input/output operations per second.

Replication: Maintaining multiple copies of data.

Partitioning: Dividing data into subsets for scale or access efficiency.

Consistency: Rules governing what values reads may observe.

RPO: Maximum tolerable data loss expressed as a time interval.

RTO: Maximum tolerable service restoration time.

Relational database: Table-based database supporting relationships and SQL.

NoSQL: Family of non-relational database models such as key-value and document stores.

Graph database: Database organized around vertices and relationships.

Cache: Temporary, fast-access copy used to reduce latency or backend load.

Data lake: Scalable repository retaining data in varied formats and processing states.

Lakehouse: Governed data platform combining lake storage with table-management features.

Data lineage: Record of data origin, transformations and destinations.

Data sovereignty: Legal and policy control associated with data location and processing.

Idempotency: Property allowing safe repetition of an operation without unintended duplication.

Review Questions

Short Answer Questions

1. What is the principal difference between object and block storage?
2. Define RPO and RTO.
3. Why can a cache not normally be treated as the system of record?
4. State two reasons for using a data catalogue.
5. What is a hot partition?

Long Answer Questions

1. Compare object, block and file storage for a media platform, including performance, sharing and lifecycle considerations.
2. Explain how relational, NoSQL, graph and analytical databases address different workload characteristics.
3. Discuss synchronous and asynchronous replication and their effects on consistency, latency and recovery.
4. Describe the architecture of a governed data lake or lakehouse from ingestion through consumption.
5. Explain how sovereignty, retention, encryption and deletion requirements influence replication and backup design.

Analytical / Application Questions

1. A retail system needs payment integrity, product search, session storage and daily sales analysis. Propose suitable data technologies for each workload and justify the boundaries between them.
2. A streaming pipeline produces duplicate and late events during network disruption. Design controls for replay, deduplication, ordering and reconciliation.
3. An organization wants to replicate a sensitive dataset across three regions for availability. Analyze the durability, correlated-failure, sovereignty and governance risks before approving the design.

Multiple Choice Questions with Answers

Question 1

Which storage type is most appropriate for immutable video archives accessed through an API?

A. Object storage

B. Block storage

C. Shared file storage

D. CPU cache

Correct Answer: A. Object storage

Question 2

Which database model is generally best suited to enforcing multi-record payment transactions and referential constraints?

A. Key-value database

B. Relational database

C. Graph database

D. Cache

Correct Answer: B. Relational database

Question 3

A cache records 900 hits and 100 misses in an interval. Using h = H ÷ (H + M), what is the hit ratio?

A. 0.09

B. 0.10

C. 0.90

D. 1.11

Correct Answer: C. 0.90

Question 4

What is the principal risk of using a poorly chosen partition key?

A. All data becomes automatically encrypted

B. Traffic and storage may become unevenly concentrated

C. Every transaction becomes strongly consistent

D. Backups become unnecessary

Correct Answer: B. Traffic and storage may become unevenly concentrated

Question 5

Which statement best describes RPO?

A. The maximum acceptable time to restore service

B. The maximum acceptable amount of data loss measured in time

C. The number of replicas required for durability

D. The average query response time

Correct Answer: B. The maximum acceptable amount of data loss measured in time

Question 6

What is a defining characteristic of a data lakehouse?

A. It permits only relational tables

B. It combines lake-scale storage with governed table-management features

C. It stores only temporary cache entries

D. It eliminates the need for metadata

Correct Answer: B. It combines lake-scale storage with governed table-management features

Question 7

Which pipeline technique most directly reduces unintended duplication after message retries?

A. Idempotent writes using stable identifiers

B. Deleting all failed messages

C. Disabling schema validation

D. Using a larger object name

Correct Answer: A. Idempotent writes using stable identifiers

Question 8

Why might a columnar analytical database outperform a row-oriented operational database for a report aggregating a few columns across billions of rows?

A. It can read and compress only relevant columns efficiently

B. It guarantees zero network latency

C. It removes the need for access controls

D. It always provides stronger transactional isolation

Correct Answer: A. It can read and compress only relevant columns efficiently

Question 9

Which statement about asynchronous replication is correct?

A. It always eliminates replication lag

B. It can acknowledge writes before all replicas have received them

C. It guarantees no data loss after primary failure

D. It requires no network connection

Correct Answer: B. It can acknowledge writes before all replicas have received them

Question 10

Which practice best addresses data sovereignty in a replicated architecture?

A. Replicate to every available region by default

B. Document and control storage, processing and backup locations

C. Put sensitive identifiers in filenames

D. Use a cache instead of governance

Correct Answer: B. Document and control storage, processing and backup locations