Distributed and Cloud Database Systems

Introduction

A distributed database stores and manages data across multiple networked computers while presenting applications with a coordinated data service. Cloud database systems extend this idea by using elastic, virtualized infrastructure delivered through managed services or application platforms. These systems are central to modern artificial-intelligence applications, e-commerce, financial services, telecommunications, and Internet-scale analytics because a single server may not provide sufficient capacity, availability, geographic reach, or fault tolerance.

Distribution, however, introduces costs. Network communication is slower and less reliable than memory or local storage; clocks are imperfect; machines fail independently; and a transaction that spans several sites is more difficult to coordinate than a local transaction. The central design problem is therefore not simply how to store more data, but how to balance correctness, availability, latency, resilience, scalability, and cost.

This chapter examines fragmentation, replication, distributed queries and transactions, consensus, consistency models, CAP and PACELC, and cloud database deployment choices. It also considers serverless systems, scaling strategies, governance, and practical selection criteria.

Learning Objectives

After studying this chapter, you should be able to:

1. Explain the architecture and motivations of distributed and cloud databases.

2. Evaluate horizontal, vertical, and derived fragmentation.

3. Compare replication strategies and identify their consistency and availability consequences.

4. Analyze distributed query execution and distributed transaction protocols.

5. Explain consensus and distinguish it from ordinary replication.

6. Apply CAP and PACELC to architectural decisions without treating them as simplistic product labels.

7. Compare managed cloud databases, serverless databases, and self-managed deployments.

8. Recommend scaling, consistency, and cost–performance strategies for realistic applications.

1. Foundations of Distributed Database Systems

A distributed database is a logically integrated collection of data stored at more than one networked site. A distributed database management system (DDBMS) coordinates storage, query processing, transactions, security, and recovery across those sites. Ideally, users need not know where a data item is located; this property is called distribution transparency.

Important forms of transparency include:

• Location transparency: applications refer to logical data rather than a physical server.
• Fragmentation transparency: applications need not know that a relation has been divided.
• Replication transparency: applications need not manage multiple copies directly.
• Failure transparency: the system masks some site or network failures.
• Concurrency transparency: simultaneous operations appear to follow the system’s stated isolation rules.

A distributed system can be homogeneous, in which sites use compatible DBMS technology, or heterogeneous, in which different database products, schemas, and data models are integrated. Cloud systems commonly combine relational databases, document stores, key-value stores, object storage, streaming platforms, and analytical warehouses.

Distribution is adopted for several reasons:

• Capacity: data and workload can exceed one machine’s storage or compute resources.
• Availability: replicas can continue serving requests after a component fails.
• Geographic performance: data can be placed near users.
• Organizational autonomy: branches or departments may retain local control.
• Elasticity: resources can expand or contract with demand.
• Workload specialization: transactional and analytical workloads can use different engines.

The costs include coordination traffic, operational complexity, consistency delays, cross-region transfer charges, more difficult debugging, and a larger security boundary.

2. Fragmentation and Data Placement

Fragmentation divides a relation into smaller pieces called fragments. A correct fragmentation scheme should satisfy three classical properties:

1. Completeness: every original tuple and attribute needed by the design is represented in the fragments.

2. Reconstruction: the original relation can be reconstructed using relational operations.

3. Disjointness, where required: fragments should not duplicate the same data unless intentional replication is used.

2.1 Horizontal Fragmentation

Horizontal fragmentation divides rows. For relation CUSTOMER, fragments might be defined as:

• CUSTOMER_EU = rows where region = 'EU'
• CUSTOMER_ASIA = rows where region = 'ASIA'

For a simple range or predicate partition, reconstruction uses UNION. If predicates are mutually exclusive and collectively exhaustive, the fragments are disjoint and complete. Horizontal fragmentation is useful when regional applications mostly access local customers.

Derived horizontal fragmentation divides a relation according to the fragmentation of a related relation. For example, ORDERS can be placed with CUSTOMER fragments by following the customer_id relationship. This can reduce distributed joins, but it may duplicate or move data when customers change region.

2.2 Vertical Fragmentation

Vertical fragmentation divides columns. A CUSTOMER table might be divided into:

• CUSTOMER_PUBLIC(customer_id, name, region)
• CUSTOMER_PRIVATE(customer_id, credit_limit, risk_score)

The primary key is included in each fragment so that the original relation can be reconstructed through a join. Vertical fragmentation can improve security and reduce I/O, but queries requiring columns from several fragments incur communication and join costs.

2.3 Hybrid Fragmentation and Placement

Hybrid fragmentation applies horizontal and vertical operations in sequence. Fragment design should reflect workload locality, data sensitivity, update frequency, and expected growth. A placement decision then maps fragments to sites or cloud regions.

A useful design question is whether to colocate related data. Colocation reduces network joins and transaction coordination, but it can create hot spots and reduce geographic flexibility. Splitting data improves parallelism and fault isolation, but increases distributed operations.

Partitioning keys require particular care. A high-cardinality, well-distributed key can reduce skew, whereas a monotonically increasing key may concentrate new writes on one partition. Hash partitioning usually balances load but does not preserve range locality. Range partitioning supports range queries and archival policies but can create uneven partitions.

3. Replication and Its Trade-offs

Replication maintains multiple copies of data. A replica may improve read performance, availability, geographic latency, and disaster recovery. It also consumes storage and network bandwidth and creates the problem of keeping copies sufficiently synchronized.

3.1 Replication Strategies

Synchronous replication acknowledges a write only after required replicas have persisted it. It can provide strong durability and a clear failover point, but latency is bounded by the slowest participating replica and network path.

Asynchronous replication acknowledges the write before all replicas receive it. It typically offers lower write latency and greater geographic reach, but replicas may lag. A failover can lose recently acknowledged writes unless the system has additional safeguards.

Primary-backup replication directs writes to a leader or primary and propagates them to followers. Leaderless approaches allow several replicas to accept operations and use quorum or conflict-resolution mechanisms. Multi-primary replication permits writes in multiple locations but makes conflict detection and resolution especially important.

3.2 Quorum Reasoning

In a replicated system with N replicas, a write quorum W and read quorum R may be selected. If R + W > N, a read quorum and a write quorum overlap, which can support reading a recent value under specific protocol assumptions. If W > N/2, two successful writes cannot both have disjoint write quorums. These conditions alone do not guarantee linearizability: version ordering, failure handling, read repair, and membership rules also matter.

For example, with N = 3, W = 2, and R = 2, every successful read and write share at least one replica. This is stronger than using W = 1 and R = 1, but it can reduce availability when two replicas cannot be reached. Quorum systems must therefore be evaluated together with failure models and consistency guarantees.

3.3 Conflict Resolution

Asynchronous or multi-primary systems may receive concurrent updates. Resolution may use timestamps, logical versions, application-defined rules, last-write-wins, or merge functions. Last-write-wins is simple but can silently discard valid updates, especially when clocks are inaccurate or updates are semantically different. Domain-specific merging is often safer, such as adding independent shopping-cart items rather than replacing an entire cart.

4. Distributed Query Processing

A distributed query processor transforms a declarative query into an execution plan involving multiple sites. The main objectives are correctness, low response time, and low resource consumption.

The process commonly includes:

1. Query decomposition into relational operations.

2. Data localization, replacing global relations with their fragments.

3. Global optimization, including join order, site selection, and parallelism.

4. Local optimization at each site.

5. Distributed execution and result assembly.

Communication cost is frequently more important than local CPU cost. If a relation of size S bytes must be transferred over a link with effective bandwidth B bytes per second, the idealized transmission time is approximately S/B, excluding latency, protocol overhead, contention, and serialization costs. This expression is an estimate, not a complete performance model.

A distributed join can use several strategies:

• Ship one relation to the site containing the other relation.
• Ship both relations to a selected site.
• Send only projected and filtered columns.
• Use a semijoin to send join keys first, reducing later transfer.
• Partition both inputs by join key and perform a repartition join.

Suppose CUSTOMER has 10 million rows but a predicate selects only 100,000 customers, and ORDERS is stored at another site. Applying the predicate and projecting only customer_id before transfer may be substantially cheaper than sending the complete CUSTOMER table. The optimizer must nevertheless consider selectivity estimates, network bandwidth, data skew, and the cost of materializing intermediate results.

Cloud query engines also exploit parallel scans, distributed hash joins, columnar storage, caching, and elastic worker pools. These mechanisms improve throughput, but poorly chosen partition keys can cause one worker to process most of the data while others remain idle.

5. Distributed Transactions and Recovery

A distributed transaction accesses data at multiple sites and should preserve the transaction properties of atomicity, consistency, isolation, and durability. The difficulty is achieving a single outcome when participants and communication links can fail independently.

5.1 Two-Phase Commit

Two-phase commit (2PC) uses a coordinator and participants.

Phase 1, prepare:

1. The coordinator asks participants whether they can commit.

2. Each participant records enough information to commit or roll back and replies prepared or unable.

Phase 2, decision:

1. If all participants are prepared, the coordinator records and sends commit.

2. Otherwise, it records and sends abort.

3. Participants record the decision and release resources.

2PC provides atomic commitment under stated logging and recovery assumptions, but it can block. A participant that is prepared but cannot contact the coordinator may be unable to decide safely whether to commit or abort. Long-lived distributed transactions can also hold locks or versions and increase latency.

5.2 Alternatives and Practical Patterns

Three-phase commit attempts to reduce certain blocking situations but adds messages and assumptions; it is not a universal solution to network partition uncertainty. Consensus-based commit protocols can improve fault tolerance in particular architectures, but they still incur coordination costs.

Many cloud applications avoid large distributed ACID transactions by using local transactions plus an outbox pattern, sagas, idempotent operations, and compensating actions. A saga divides a business process into local transactions. If a later step fails, compensating actions semantically undo earlier effects. Compensation is not identical to rollback: an email already sent, for example, cannot be literally unsent.

Idempotency is essential for retryable distributed operations. An operation is idempotent if repeating it has the same intended effect as performing it once. A unique request identifier can prevent duplicate payment or order creation when a client retries after a timeout.

6. Consensus and Coordination

Consensus is the problem of causing distributed processes to agree on a value despite certain failures. A consensus protocol is usually described using agreement, validity, and termination properties, subject to a specified failure and timing model.

Raft and Paxos are families of consensus algorithms used to replicate a log or coordinate a cluster. In a typical leader-based system, a leader proposes ordered entries, followers persist them, and an entry becomes committed after the protocol’s quorum condition is met. Consensus is not the same as replication: replication copies data, while consensus provides an agreement mechanism for ordering or deciding operations under failures.

Consensus in an asynchronous system with even one crash failure cannot guarantee termination in every possible execution; this is the essence of the FLP impossibility result. Practical systems obtain progress by using timing assumptions, failure detectors, leases, randomized behavior, or a distinction between normal operation and pathological executions.

Consensus has practical costs. Cross-region consensus increases write latency, and a quorum may be unavailable during a partition. It is therefore appropriate for metadata, leader election, configuration, and strongly ordered records, but not every workload needs global consensus for every read or write.

7. Consistency Models

Consistency describes what values and orderings clients may observe. It is distinct from isolation, which describes how concurrent operations within transactions interact, although the two are related.

• Eventual consistency: if updates stop and communication continues, replicas eventually converge. Temporary stale reads are allowed.
• Read-your-writes consistency: a client sees its own successful writes in later reads.
• Monotonic reads: a client does not observe data moving backward to an older state.
• Causal consistency: causally related operations are observed in causal order, while concurrent operations may be seen in different orders.
• Sequential consistency: operations appear in one sequential order consistent with each client’s program order.
• Linearizability: each operation appears to take effect at one instant between invocation and response, respecting real-time precedence.
• Serializable transaction isolation: the outcome is equivalent to some serial execution of committed transactions, although it does not automatically define real-time behavior for all individual reads.

A social-media “like” counter may tolerate temporary staleness, whereas an account withdrawal generally requires stronger safeguards. Consistency should be selected per operation or data item where possible, rather than imposed uniformly at maximum cost.

8. CAP and PACELC

The CAP theorem concerns a distributed data system in the presence of a network partition. It states that a system cannot simultaneously guarantee a partition-tolerant form of consistency and availability in all partitioned executions. Since partitions are possible in distributed systems, a design must determine which behavior it prefers during a partition.

In this context, consistency means that operations observe a single coherent result according to a specified strong model, and availability means that every request to a non-failing node receives a response. CAP does not say that a system can never be consistent and available during normal operation. It also does not mean that systems are permanently labeled “CP” or “AP” for every operation.

PACELC extends the discussion: if there is a Partition, choose between Availability and Consistency; Else, during normal operation, choose between Latency and Consistency. Strong cross-region consistency can increase latency even when no failure exists. An eventually consistent geo-replicated service may offer lower latency but expose stale data.

These models are decision frameworks, not substitutes for reading product documentation. A database can offer configurable consistency, different guarantees for different APIs, or strong consistency within a region and weaker behavior across regions.

9. Cloud Database Architectures

Cloud databases are commonly consumed in three ways.

Self-managed virtual machines provide extensive control over database versions, extensions, topology, and tuning. The organization remains responsible for patching, backups, failover, monitoring, and capacity planning.

Managed database services automate many operational tasks, such as provisioning, backups, software patching, replication, encryption integration, and monitoring. They reduce undifferentiated operational work but may restrict configuration and impose service-specific limits, pricing, and migration costs.

Serverless databases automatically allocate capacity or compute in response to requests, often separating storage from compute. They can be attractive for intermittent or unpredictable workloads because idle capacity may be reduced. Cold starts, connection-management limits, burst quotas, variable latency, and per-request pricing must be considered. “Serverless” does not mean that servers do not exist; it means that their provisioning is abstracted from the customer.

Cloud database choices include relational managed services, distributed SQL systems, document databases, key-value stores, wide-column systems, graph databases, time-series databases, and analytical warehouses. Polyglot persistence can match each workload to an appropriate engine, but it increases data movement, governance complexity, and operational skills requirements.

10. Scaling and Cost–Performance Trade-offs

Vertical scaling increases the resources of one node. It is operationally simple and often preserves transaction semantics, but hardware limits and single-node failure risks remain.

Horizontal scaling adds nodes. It can provide large capacity and fault isolation, but requires partitioning, routing, rebalancing, and often distributed coordination. Read replicas scale read throughput but do not necessarily scale writes. Caching reduces database load but introduces invalidation and staleness concerns.

Common scaling patterns include:

• Read/write separation using a primary and replicas.
• Sharding by tenant, geography, or hash key.
• Time-based partitioning for event and log data.
• CQRS, separating command and query models.
• Materialized views for repeated analytical access.
• Tiered storage and archival for older data.
• Autoscaling based on workload metrics.

Cost should be evaluated as total cost of ownership, not only hourly compute price. Relevant components include storage, provisioned capacity, requests, backups, replicas, cross-zone and cross-region transfer, observability, support, migration, engineering labor, and the financial cost of downtime or data loss.

A low-cost architecture that causes frequent outages may be more expensive overall than a moderately priced highly available service. Conversely, overprovisioning a globally replicated cluster for a small, low-risk workload wastes resources. Benchmarking should use realistic data volume, skew, concurrency, query mix, failure scenarios, and recovery requirements.

11. Case Study: A Global Learning Platform

The following is an illustrative case.

Background

A learning platform serves students in several continents. It stores user profiles, course enrollments, quiz attempts, payment records, video metadata, and personalized recommendations generated by machine-learning models.

Problem

A single relational database produced high latency for distant users and became difficult to scale during examination periods. The team considered globally replicating all tables, but payment correctness and recommendation freshness had different requirements.

Analysis

Payment authorization and enrollment changes require strong transactional guarantees. Quiz-event ingestion must absorb bursts and can tolerate delayed analytical visibility. Video files are large and should not be stored in the transactional database. Recommendations can use slightly stale features, provided that model versions and feature provenance are recorded.

Approach

The platform adopts a managed relational database for payments and enrollment, with synchronous high-availability replication within a region and a tested disaster-recovery replica in another region. Enrollment is partitioned by institution where practical, while payment records remain within approved data-governance boundaries.

An event stream receives quiz attempts using idempotent event identifiers. Local writes are committed quickly, and an outbox mechanism publishes reliable change events. A warehouse consumes these events for analytics and model training. Recommendation features are stored in a scalable key-value or feature-store system with documented freshness limits. Object storage serves videos through a content-delivery network.

The application routes users to nearby read replicas for catalog and progress views, but routes enrollment and payment decisions to the authoritative transactional region. During a regional failure, the disaster-recovery plan prioritizes restoration of authoritative writes rather than allowing uncontrolled multi-primary payment updates.

Outcome

The design improves regional read latency and absorbs examination-period ingestion without placing all workloads on one database. It does not eliminate complexity: event duplication, replica lag, recovery-point objectives, and cross-system reconciliation must be monitored.

Learning Points

1. One consistency model rarely fits every data item.

2. Data placement should follow business boundaries and access patterns.

3. Machine learning benefits from event-driven pipelines but does not remove transactional requirements.

4. A recovery design must specify acceptable data loss and recovery time, not merely claim high availability.

5. Managed services reduce operational work but do not remove architectural responsibility.

12. Governance, Security, and Ethical Considerations

Distributed cloud systems expand the locations and organizations that may process data. Architects must address data residency, retention, encryption in transit and at rest, key management, access control, auditability, backup deletion, and incident response. Replicas and backups may persist data after an application record is deleted, so deletion policies must include derived copies and archives.

Privacy regulations may restrict cross-border transfer or require access, correction, and erasure processes. Data minimization and purpose limitation are especially important for AI systems, where replicated training data and feature stores can unintentionally preserve sensitive information. Access logs should be protected because they can reveal user behavior.

Service-level objectives should distinguish availability, latency, durability, recovery-time objective, and recovery-point objective. Vendors’ service-level agreements may provide credits rather than restore business continuity; independent backups and recovery tests remain necessary.

13. Best Practices and Contemporary Developments

Effective distributed database engineering begins with explicit requirements: workload shape, transaction boundaries, consistency needs, geographic scope, failure tolerance, compliance, and budget. Teams should document the data ownership model, partition keys, replication topology, conflict policy, and recovery procedures.

Use observability that measures query latency, replica lag, transaction aborts, lock waits, partition skew, hot keys, storage growth, network transfer, and cost by workload. Test failover, partition behavior, backup restoration, rebalancing, and schema migration rather than relying solely on diagrams.

Contemporary systems increasingly separate storage and compute, use disaggregated architectures, offer programmable consistency, and combine transactional databases with streaming and analytical systems. HTAP designs attempt to support transactional and analytical processing in related platforms, although resource contention and workload isolation remain challenges. AI-assisted database administration can help detect anomalies, recommend indexes, forecast capacity, and summarize incidents; human review is still necessary because automated recommendations may optimize a local metric while violating compliance or consistency requirements.

Chapter Summary

Distributed databases place data and computation across networked sites to obtain capacity, availability, locality, and elasticity. Fragmentation divides data by rows, columns, or both; good fragmentation is complete, reconstructable, and aligned with access patterns. Replication improves resilience and read performance but introduces lag, conflicts, and coordination costs. Synchronous replication favors stronger durability, while asynchronous replication favors lower latency and greater geographic flexibility.

Distributed query processing must minimize communication through localization, filtering, projection, join planning, and parallel execution. Distributed transactions use mechanisms such as two-phase commit, but blocking and latency motivate local transactions, sagas, outbox patterns, and idempotent operations. Consensus establishes agreement under a failure model and is a foundation for replicated logs and coordination.

Consistency ranges from eventual and session guarantees to causal consistency, sequential consistency, and linearizability. CAP explains choices during network partitions, while PACELC highlights the normal-case latency–consistency trade-off. Cloud databases may be self-managed, managed, or serverless; each has distinct control, operational, scalability, and pricing characteristics. Sound architecture matches consistency and placement to business value, measures total cost, tests failure recovery, and treats security, privacy, governance, and observability as integral design requirements.

Key Terms

Distributed database: A logically integrated database whose data is stored across networked sites.

DDBMS: A system that coordinates data management and processing across distributed sites.

Fragmentation: Dividing a relation into smaller logical pieces.

Horizontal fragmentation: Partitioning a relation by rows.

Vertical fragmentation: Partitioning a relation by columns while retaining reconstruction keys.

Replication: Maintaining multiple copies of data.

Quorum: A required subset of replicas that participates in a read, write, or decision.

Replica lag: The delay before a replica reflects an update accepted elsewhere.

Distributed query: A query whose data or processing spans multiple sites.

Two-phase commit: A coordinator-based atomic commitment protocol with prepare and decision phases.

Saga: A sequence of local transactions coordinated with compensating actions.

Consensus: Agreement among distributed processes despite specified failures.

Consistency model: A formal description of values and operation order visible to clients.

Linearizability: A guarantee that operations appear instantaneous and respect real-time ordering.

CAP theorem: A result concerning consistency, availability, and partition behavior.

PACELC: A framework adding normal-case latency–consistency trade-offs to CAP reasoning.

Sharding: Horizontal distribution of data across partitions or nodes.

Serverless database: A database service whose infrastructure provisioning is largely abstracted and dynamically managed.

Recovery-point objective: The maximum acceptable amount of data loss measured in time.

Recovery-time objective: The target time for restoring service after disruption.

Review Questions

Short Answer Questions

1. What are the three classical correctness properties of fragmentation?

2. Distinguish synchronous from asynchronous replication.

3. Why can two-phase commit block?

4. What does linearizability guarantee?

5. Why does serverless computing not mean that servers are absent?

Long Answer Questions

1. Explain horizontal, vertical, derived, and hybrid fragmentation, including appropriate use cases.

2. Compare primary-backup, multi-primary, and leaderless replication with respect to latency, availability, and conflict management.

3. Describe distributed query processing and explain why communication cost often dominates local computation.

4. Explain two-phase commit and discuss why sagas and idempotency are useful alternatives for cloud applications.

5. Compare self-managed, managed, and serverless cloud databases using operational control, scalability, resilience, and cost.

Analytical / Application Questions

1. A three-replica system uses a read quorum of two and a write quorum of two. Explain what overlap property this provides and identify additional assumptions needed before claiming linearizable reads.

2. Design a consistency and placement strategy for a global payment and recommendation platform. Justify which data requires strong consistency and which data may tolerate staleness.

3. A company wants to reduce cost by using asynchronous cross-region replication. Analyze the likely effects on latency, recovery-point objectives, failover behavior, compliance, and customer-visible correctness.

Multiple Choice Questions with Answers

Question 1

Which fragmentation method divides a relation by rows?

A. Horizontal fragmentation

B. Vertical fragmentation

C. Replication

D. Serialization

Correct Answer: A. Horizontal fragmentation

Question 2

With N = 3 replicas, which quorum pair provides overlapping read and write quorums under the usual quorum condition?

A. R = 1, W = 1

B. R = 1, W = 2

C. R = 2, W = 2

D. R = 1, W = 0

Correct Answer: C. R = 2, W = 2

Question 3

What is the principal blocking problem of two-phase commit?

A. A participant may be unable to decide after preparing if it loses contact with the coordinator

B. It cannot write a transaction log

C. It prevents all local transactions

D. It guarantees duplicate commits

Correct Answer: A. A participant may be unable to decide after preparing if it loses contact with the coordinator

Question 4

Which consistency model respects real-time ordering of non-overlapping operations?

A. Eventual consistency

B. Linearizability

C. Best-effort consistency

D. Read-your-writes consistency only

Correct Answer: B. Linearizability

Question 5

In distributed query processing, why is projection often pushed before data transfer?

A. To increase the number of columns sent

B. To reduce communication volume

C. To disable indexing

D. To force serial execution

Correct Answer: B. To reduce communication volume

Question 6

What is the main purpose of a consensus protocol such as Raft?

A. Compressing database backups

B. Selecting indexes automatically

C. Helping distributed processes agree on an ordered decision despite specified failures

D. Replacing all transaction isolation

Correct Answer: C. Helping distributed processes agree on an ordered decision despite specified failures

Question 7

According to CAP, what trade-off is unavoidable when a network partition occurs?

A. Storage versus indexing

B. Consistency versus availability

C. CPU versus disk size

D. Encryption versus compression

Correct Answer: B. Consistency versus availability

Question 8

What additional trade-off does PACELC emphasize during normal operation?

A. Latency versus consistency

B. Storage versus durability

C. Security versus authentication

D. Partitioning versus backup

Correct Answer: A. Latency versus consistency

Question 9

Which pattern helps prevent duplicate effects when a distributed request is retried?

A. Idempotency using a unique request identifier

B. Removing transaction logs

C. Using random partition keys for every retry

D. Disabling acknowledgements

Correct Answer: A. Idempotency using a unique request identifier

Question 10

Which statement best describes a serverless database?

A. It has no physical machines

B. It cannot scale automatically

C. Infrastructure provisioning is largely abstracted and capacity may adjust dynamically

D. It always provides linearizability

Correct Answer: C. Infrastructure provisioning is largely abstracted and capacity may adjust dynamically