Distributed Systems, Scalability and Reliability

Introduction and Context

Cloud-based AI services rarely operate as a single program on one machine. A production inference platform may receive requests from many regions, retrieve features from distributed stores, invoke several models, publish events, and record results for later analysis. Training pipelines may process petabytes of data across clusters, while online systems must respond within milliseconds. These requirements make distributed-systems engineering a central part of cloud computing for AI.

A distributed system is a collection of independent computers that cooperate over a network and appear to users as one system. The computers may fail independently, communication may be delayed or lost, and different components may observe events in different orders. Consequently, distributed design is not simply a matter of adding more servers. It requires explicit decisions about data placement, coordination, consistency, failure handling, and operational visibility.

This chapter examines partitioning, replication, consensus, queues, event-driven processing, eventual consistency, CAP-related trade-offs, fault tolerance, and stateless design. The emphasis is on applying these ideas to scalable and reliable AI services.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain why distributed AI systems require partitioning, replication, coordination, and fault tolerance.

2. Select appropriate scaling strategies for inference, feature serving, data processing, and model-management workloads.

3. Compare strong, causal, and eventual consistency in practical cloud applications.

4. Explain the CAP perspective and distinguish it from simplified slogans about distributed systems.

5. Design queue-based and event-driven workflows with retries, idempotency, and dead-letter handling.

6. Apply stateless design, health checks, graceful degradation, and observability to reliable services.

7. Analyze trade-offs among latency, availability, cost, correctness, and operational complexity.

1. Distributed-System Foundations

1.1 Why distribution is necessary

Distribution is used for several related reasons:

• Capacity: A single machine may not provide enough CPU, memory, storage, or accelerator capacity.
• Elasticity: Cloud resources can be added or removed as demand changes.
• Geographic proximity: Regional deployment can reduce network latency and support data-locality requirements.
• Availability: Multiple instances can continue serving when one fails.
• Specialization: Separate services can handle inference, feature retrieval, authorization, logging, and model management.
• Throughput: Independent work can be processed concurrently.

For example, an image-classification service may place a load balancer in front of multiple inference replicas. A feature store may distribute records across database partitions. A training pipeline may divide a large dataset into shards and process them in parallel.

1.2 Sources of complexity

In a centralized program, memory operations are usually fast and failure behavior is relatively predictable. In a distributed system, a network call can fail, time out, or succeed without the caller receiving the response. Clocks are not perfectly synchronized, and two replicas may temporarily contain different values. A component that is slow may be indistinguishable from one that has failed unless timeouts and health signals are used carefully.

The system therefore cannot assume that all participants share one instantaneous state. Instead, it must define acceptable guarantees. Examples include:

• Every read sees the latest committed value.
• A read may temporarily return an older value, but replicas converge.
• A request is processed at least once, with duplicate effects prevented.
• A prediction is served only if the feature version matches the model version.

These are engineering contracts, not merely implementation details.

2. Scalability Models and Capacity Reasoning

Scalability describes how effectively a system handles increasing demand or resources. It is different from performance: performance concerns behavior at a particular load, whereas scalability concerns behavior as load changes.

2.1 Vertical and horizontal scaling

Vertical scaling increases the resources of one machine, such as CPU, RAM, or GPU memory. It is straightforward but limited by hardware size and often creates a large failure domain.

Horizontal scaling adds more machines or service instances. It supports elasticity and fault isolation, but introduces coordination, networking, data distribution, and consistency problems.

AI workloads often use both. A GPU inference replica may be vertically optimized with batching and model quantization, while the service is horizontally replicated across nodes.

2.2 Throughput and utilization

For a stable service, the long-run arrival rate must not exceed sustainable service capacity. If requests arrive at rate λ requests per second and each replica can sustainably process μ requests per second, then a simple capacity estimate for N identical replicas is:

Capacity = Nμ

The estimate assumes comparable replicas, balanced traffic, and sufficient network, storage, and coordination capacity. In practice, systems reserve headroom because performance often deteriorates near saturation. A design targeting 100% utilization has little tolerance for bursts, failures, or background work.

A useful reliability-oriented estimate is the number of replicas required after allowing for one replica failure:

Nμ(1 − r) ≥ λ_peak

where r is the fraction of capacity lost in the failure scenario, λ_peak is the planned peak arrival rate, and the inequality assumes the remaining replicas can absorb demand. This is a planning approximation, not a guarantee; correlated failures and workload variability must also be considered.

2.3 Latency and tail behavior

Average latency can conceal serious user experiences. The 99th-percentile latency is the value below which 99% of observed requests complete. Distributed calls amplify tail latency because a request may wait for the slowest of several dependencies. Reducing average latency alone may therefore fail to improve the user-visible tail.

Common techniques include bounded timeouts, parallel independent calls, caching, request hedging where appropriate, load shedding, and reducing dependency count. Hedging must be used carefully because duplicate requests can increase load and cause a feedback loop.

3. Partitioning and Sharding

Partitioning divides data or work among nodes. A partition is a logical subset, while sharding often refers specifically to distributing database data across machines.

3.1 Partitioning strategies

Range partitioning assigns contiguous key ranges to nodes. It supports range queries but can create hotspots when recent or popular keys cluster in one range.

Hash partitioning applies a hash function to a key and maps the result to a partition. It distributes point lookups relatively evenly but makes range queries more difficult.

Consistent hashing organizes hash values around a logical ring and assigns keys to nearby nodes. Adding or removing a node can move fewer keys than a naive modulo scheme, although virtual nodes and rebalancing are usually needed for good distribution.

Directory-based partitioning uses a mapping service to identify the partition for each key. It permits flexible placement but makes the directory an important dependency.

3.2 Choosing a partition key

A good partition key distributes traffic and storage while preserving the access patterns required by the application. In an AI feature store, user ID may support user-centric lookups, whereas a timestamp may support time-window analysis but create a current-time hotspot. A composite key such as tenant ID plus entity ID may provide isolation and distribution.

Partitioning should consider:

• expected read and write frequency;
• record size and growth rate;
• query patterns;
• tenant isolation;
• data residency requirements;
• rebalancing cost;
• skew and “hot keys.”

A partitioned model registry, for example, might use model name and version as a logical key, while large artifact files are stored in object storage rather than in the metadata database.

3.3 Partition tolerance and operational issues

Partitioning creates cross-partition operations. A query spanning many partitions may require fan-out, increasing latency and failure probability. Transactions across partitions are more complex than local transactions. Rebalancing can consume bandwidth and compete with application traffic. Systems should therefore colocate data that is commonly accessed together and avoid unnecessary distributed transactions.

4. Replication and Data Consistency

Replication stores multiple copies of data or maintains multiple service instances. It improves availability and read capacity, but replicas can diverge temporarily or permanently if updates are mishandled.

4.1 Replication patterns

• Leader–follower replication: A leader accepts writes and followers copy them. It is conceptually simple, but leader failure requires election or reassignment.
• Multi-leader replication: Multiple sites accept writes. It can reduce geographic write latency but requires conflict resolution.
• Leaderless replication: Clients or coordinators communicate with several replicas. Availability can be high, but read and write quorum policies must be designed carefully.
• Synchronous replication: A write is acknowledged only after specified replicas confirm it. This can provide stronger durability but adds latency and may reduce availability during network problems.
• Asynchronous replication: The writer is acknowledged before all replicas receive the update. It reduces latency but permits replication lag and possible data loss during certain failures.

4.2 Consistency models

Strong consistency means that completed writes are observed according to a specified global order, commonly with linearizable behavior for a data object. It is appropriate for operations such as allocating a unique resource or enforcing a safety limit, though it may require coordination.

Sequential consistency requires that operations appear in one order consistent with each client’s program order, but not necessarily with real-time order across clients.

Causal consistency preserves cause-and-effect relationships. If a user updates a profile and then requests a prediction, the prediction path should not observe a state older than the update if the update causally precedes the request.

Eventual consistency permits temporary divergence but requires replicas to converge if updates stop and communication continues. It is useful for caches, recommendation features, and asynchronously replicated analytical views when temporary staleness is acceptable.

Eventual consistency does not mean random or permanently inconsistent behavior. It requires a convergence mechanism, such as last-write-wins with carefully defined timestamps, version vectors, conflict-free replicated data types, or application-specific conflict resolution. Last-write-wins is convenient but can discard valid updates when clocks or timestamp policies are unsuitable.

4.3 Consistency in AI systems

A feature used for training and serving may exist in several versions. If a prediction uses a stale feature while the model expects a newer schema, the result may be inaccurate even though the database is technically available. AI systems therefore need semantic consistency, including feature definitions, model versions, preprocessing code, and labels.

Strong consistency is generally more important for model authorization, experiment assignment, and billing. Eventual consistency may be acceptable for dashboards, popularity counts, or asynchronously refreshed recommendations. The appropriate choice depends on the consequence of stale data.

5. CAP and Related Trade-offs

The CAP perspective states that, during a network partition, a distributed data system cannot simultaneously guarantee both availability and strong consistency for all operations. The three properties are commonly described as:

• Consistency: Every successful operation observes a result consistent with the system’s defined single-copy semantics.
• Availability: Every request to a non-failing node receives a response, although the response may not reflect the newest data.
• Partition tolerance: The system continues operating despite communication failure between nodes.

Because network partitions are possible in distributed environments, practical systems must tolerate them and choose how operations behave during the partition. A consistency-oriented system may reject or delay writes until it can coordinate. An availability-oriented system may accept operations on both sides and reconcile them later.

CAP is not a classification saying that a system permanently possesses only two properties. It describes a trade-off under a partition and concerns a particular data operation or subsystem. It also does not eliminate other dimensions such as latency, durability, recoverability, security, and cost. PACELC-style reasoning extends the discussion by asking what a system chooses between latency and consistency when there is no partition.

For an AI recommendation cache, serving a slightly stale result may be preferable to failing. For model rollout authorization, serving an obsolete or unauthorized model may be unacceptable; the service should favor correctness and possibly fail closed.

6. Consensus and Coordination

Consensus enables distributed participants to agree on a value or an ordered sequence despite failures. Consensus protocols such as Raft and Paxos are used in coordination services, replicated logs, leader election, and metadata management. A typical consensus group elects a leader, replicates proposals to followers, and commits a value after an appropriate quorum acknowledges it.

Consensus is not the same as ordinary replication. Replication copies data; consensus establishes an agreed order and commitment rule in the presence of failures. Consensus also does not make arbitrary application code fault tolerant. It coordinates a carefully specified state machine or log.

A consensus group commonly requires a majority to make progress. For a group of 2f + 1 members, a majority can tolerate up to f crash failures, assuming the failure model and network conditions meet the protocol’s assumptions. This statement concerns crash tolerance, not arbitrary malicious behavior, and does not guarantee availability during every network partition.

Consensus introduces network round trips and can become a bottleneck if every high-volume event requires global coordination. Use it for metadata, leader election, configuration, and unique allocation—not automatically for every prediction request.

7. Queues, Streams, and Event-Driven Processing

A queue decouples producers from consumers. Producers place messages into a durable buffer, and consumers process them later. This absorbs bursts, permits independent scaling, and isolates temporary failures.

A work queue generally distributes tasks among competing consumers. An event stream retains an ordered sequence of events for one or more consumer groups, allowing different applications to process the same facts independently.

7.1 Delivery semantics

• At-most-once: A message may be lost, but is not intentionally redelivered. This can be suitable for low-value telemetry.
• At-least-once: A message is retried until acknowledged or otherwise resolved. Duplicates are possible and consumers must be idempotent.
• Exactly-once effect: The externally visible result occurs once, usually through transactions, deduplication, idempotency keys, or atomic state transitions. “Exactly once” is not a magical property of a transport; it depends on the complete processing path.

7.2 Reliable consumer design

A robust consumer should validate the message, record an idempotency key, perform the intended state transition, and acknowledge only after the required effect is durable. If processing fails, the message should be retried with bounded exponential backoff and jitter. Repeatedly failing messages should move to a dead-letter queue for inspection rather than block all healthy work.

Backpressure occurs when producers generate work faster than consumers can process it. Systems can respond by increasing consumers, reducing input rate, batching work, prioritizing messages, or rejecting low-value work. For GPU inference, queue-based micro-batching can improve accelerator utilization, but excessive batching increases latency.

7.3 Event-driven AI workflows

A document-processing pipeline may emit events such as `DocumentUploaded`, `TextExtracted`, and `EmbeddingCreated`. Each stage consumes an event and publishes a new event. This architecture allows independent scaling and replay, but requires schema evolution, event ordering policies, duplicate handling, and trace propagation.

Events should represent facts that have occurred rather than commands disguised as facts. Schemas should be versioned, and consumers should tolerate fields added in compatible versions. Sensitive data should not be placed in events unnecessarily; references to protected storage are often safer than copying personal data into many systems.

8. Fault Tolerance and Failure Handling

Fault tolerance is the ability to continue providing an acceptable service despite component failures. Reliability also depends on recovery time, data durability, correct behavior, and operational response.

8.1 Failure types

• Crash failure: A component stops functioning.
• Omission failure: A message or response is lost.
• Timing failure: A component responds too late.
• Network partition: Components cannot communicate reliably.
• Byzantine or malicious behavior: A component produces arbitrary or deceptive results; this is a stronger threat model than ordinary cloud service failures.
• Correlated failure: Many nominally independent components fail because of a shared zone, dependency, deployment, or configuration.

8.2 Core mechanisms

Timeouts prevent indefinite waiting. They must be chosen with knowledge of normal latency and downstream behavior.

Retries help with transient failures but can amplify overload. Use bounded retries, exponential backoff, jitter, and retry only operations that are safe to repeat.

Circuit breakers temporarily stop calls to an unhealthy dependency, allowing it to recover and protecting callers from wasting resources.

Bulkheads isolate resource pools so that one workload cannot consume all threads, connections, or GPUs.

Health checks should distinguish liveness from readiness. A liveness check asks whether a process should be restarted; a readiness check asks whether it should receive traffic. A service that is alive but disconnected from its model store may be alive yet not ready.

Graceful degradation provides a reduced service, such as a cached prediction, a simpler model, or a delayed result. The fallback must be safe and clearly observable.

Replication and failover reduce downtime, but failover must account for stale state, split-brain risk, startup time, and dependency readiness.

9. Stateless Service Design

A stateless service does not rely on local process memory for durable client state between requests. Session information is held in signed tokens, a shared store, or a client-managed mechanism. Statelessness allows any healthy instance to serve any request, simplifying load balancing, autoscaling, and replacement.

Stateless does not mean that the overall application has no state. Databases, object stores, queues, model registries, and feature stores contain state; the principle is to externalize durable state from replaceable service instances.

A stateless inference service should package or retrieve a known model version, validate input, execute inference, and return a result without depending on previous requests reaching the same process. Local caches may still be used as performance optimizations, provided correctness does not depend on them.

Sticky sessions can simplify some stateful applications but reduce balancing flexibility and complicate failover. If used, they should be justified by a clear requirement rather than adopted by default.

10. Worked Design Example: Reliable Fraud-Scoring Service

Assume a bank requires an online fraud score for 2,000 transactions per second during ordinary peaks and 3,000 transactions per second during a major sales event. Each inference replica sustainably handles 250 requests per second at the required latency. The design must tolerate loss of one replica without falling below the 3,000-request-per-second target.

If N replicas are deployed and one is lost, the remaining capacity is (N − 1) × 250. The requirement is:

(N − 1) × 250 ≥ 3,000

Therefore:

N − 1 ≥ 12, so N ≥ 13.

At least 13 replicas are required under this simplified capacity model. In practice, more may be deployed because the estimate excludes rollout overhead, uneven traffic, dependency limits, GPU failures, and headroom. Autoscaling should respond to queue depth and latency as well as CPU utilization, because GPU inference may be constrained by accelerator saturation rather than CPU load.

The request path retrieves customer and transaction features. A strongly consistent store is used for account status and fraud-rule configuration, while a feature cache may be eventually consistent if a small amount of staleness is acceptable. The scoring service is stateless and deployed across multiple failure zones. A request carries an idempotency key so that client retries do not create duplicate transaction decisions.

If the feature store becomes slow, a timeout and circuit breaker prevent all inference workers from being consumed by waiting calls. The service may use a conservative fallback model only when governance rules permit it; otherwise, it returns a controlled “manual review” decision. Events describing the decision are sent to a durable stream for audit and investigation. The event consumer is idempotent and uses a dead-letter queue for invalid records.

Model rollout uses a registry and a coordination mechanism. A model version is promoted only after validation, and the active version is distributed to replicas. Configuration changes require an auditable, strongly coordinated update, whereas analytical dashboards consume asynchronously replicated events.

The design illustrates that reliability is a system property. Replicas alone do not solve failures in feature retrieval, configuration, event publication, model loading, or downstream auditing.

11. Case Study: Illustrative Document-Analysis Platform

Background

Consider an illustrative university service that accepts scanned assignments, extracts text, generates embeddings, and provides semantic search for instructors. Submissions vary substantially in size, and demand peaks near assessment deadlines.

Problem

A single synchronous application becomes unavailable when many large files arrive. Long-running OCR tasks occupy web-server threads, repeated client requests create duplicate processing, and an updated embedding model produces vectors incompatible with the existing index.

Analysis

The workload has two distinct characteristics: short request handling and long asynchronous computation. It also has several consistency requirements. Upload metadata must be durable, processing should not be lost, and the search index must identify the model and preprocessing version used to create each vector. Immediate search availability is useful but less important than preserving submissions and maintaining auditability.

Approach and solution

The upload service is stateless and stores files in durable object storage. It writes submission metadata and publishes a processing event using an outbox pattern, in which the database change and an event record are committed together before a publisher sends the event. This reduces the risk that metadata is stored but no processing message is emitted.

OCR workers consume tasks from a queue. Each task includes a submission identifier, a versioned schema, and an idempotency key. Workers write results to a partitioned metadata store and publish an `TextExtracted` event. Embedding workers consume that event, use a versioned model, and write vectors to a separate index. Failed tasks are retried with backoff and eventually moved to a dead-letter queue.

The search API can continue serving the previous index while a new index is built. After validation, an atomic alias or coordinated configuration change directs new searches to the new version. This avoids mixing incompatible vector spaces during rollout. The system records processing status so users can distinguish “not yet processed” from “failed.”

Outcome

The architecture absorbs deadline bursts, permits OCR and embedding workers to scale independently, and prevents a slow model stage from blocking uploads. It accepts eventual consistency between upload completion and search visibility, but preserves strong requirements for submission durability and model-version metadata.

Learning points

1. Queues are useful for absorbing bursts but do not remove the need for capacity planning.

2. At-least-once delivery requires idempotent consumers.

3. Versioned AI artifacts are part of consistency management.

4. A controlled delay is often safer than returning an untraceable or incompatible result.

5. Operational status and audit records are essential for trustworthy AI services.

12. Governance, Security, and Ethical Considerations

Distributed AI services replicate data across nodes, zones, and sometimes countries. Replication and logging can therefore increase privacy exposure. Architects should apply data minimization, encryption in transit and at rest, access control, retention limits, and regional placement policies. Personal data should not be copied into queues or debug logs without a clear purpose.

Reliability mechanisms can also affect fairness and accountability. A fallback model may have different accuracy across demographic groups. A stale feature may systematically affect users whose records update more frequently. Automated retry or load shedding may prioritize some tenants over others. These effects should be evaluated through monitoring and documented service policies.

Model and configuration changes require provenance, approval, rollback procedures, and audit trails. Access to consensus-backed configuration, model promotion, and production secrets should follow least privilege. Regulatory obligations may require explainability, data residency, incident reporting, deletion support, or reproducibility of an automated decision.

13. Contemporary Developments and Best Practices

Modern platforms increasingly combine containers, managed Kubernetes or serverless execution, service meshes, distributed tracing, and policy-as-code. These tools can automate deployment and traffic management, but they do not eliminate fundamental trade-offs. A service mesh may provide retries automatically; poorly configured retries can still overload a dependency.

Best practices include:

• define availability, latency, durability, and consistency objectives before selecting technology;
• identify the failure domains of every dependency;
• partition according to access patterns and monitor skew;
• replicate critical data and test restoration, not merely backup creation;
• make asynchronous consumers idempotent;
• use timeouts, bounded retries, jitter, circuit breakers, and dead-letter handling;
• separate liveness from readiness;
• keep application instances replaceable and durable state external;
• version event schemas, models, features, and indexes;
• use tracing and correlation identifiers across asynchronous boundaries;
• test zone loss, dependency failure, backlog growth, and stale data;
• measure tail latency, queue age, replication lag, error rate, and model-specific quality;
• document which operations require strong consistency and which tolerate eventual consistency.

Chaos experiments and disaster-recovery exercises are particularly valuable because many distributed failures arise from interactions that are not visible in unit tests. Tests should include partial failure, duplicate messages, delayed messages, replay, leader change, and incompatible model deployment.

Chapter Summary

Distributed AI services rely on multiple cooperating machines to obtain capacity, elasticity, geographic reach, and resilience. Partitioning distributes data and work but introduces cross-partition operations and hotspot risks. Replication improves availability and read capacity while creating consistency and conflict-management responsibilities.

Strong, causal, and eventual consistency provide different guarantees. CAP explains the unavoidable choice between strong consistency and availability during a network partition; it does not mean that every system permanently chooses only two properties. Consensus is appropriate for coordinated metadata and ordered state transitions, but should not be placed on every high-volume request path.

Queues and event streams decouple services, absorb bursts, and support asynchronous AI pipelines. At-least-once processing is practical, but it requires idempotency, retries, dead-letter handling, and observability. Fault tolerance depends on timeouts, controlled retries, circuit breakers, bulkheads, health checks, replication, graceful degradation, and recovery testing. Stateless service design makes instances easy to scale and replace, while durable state remains in appropriate external systems.

Reliable AI engineering requires more than infrastructure availability. Model versions, feature definitions, data lineage, privacy, auditability, and prediction quality must remain coherent across distributed components. The best architecture makes trade-offs explicit and aligns technical guarantees with the consequences of errors and stale information.

Key Terms

Distributed system: A collection of independent networked computers that cooperate as one logical service.

Scalability: The ability to handle increased demand or resources effectively.

Partitioning: Dividing data or work into subsets assigned to different components.

Sharding: Distributing database data across machines or partitions.

Replication: Maintaining multiple copies or service instances for availability, durability, or capacity.

Consistency: The guarantee describing what values and operation orders readers may observe.

Eventual consistency: A model in which replicas may diverge temporarily but converge when updates cease and communication continues.

CAP perspective: A framework describing the trade-off between consistency and availability during a network partition.

Consensus: Agreement among distributed participants on a value or ordered sequence despite specified failures.

Quorum: A sufficient subset of participants whose agreement is required for an operation or decision.

Queue: A buffer that decouples message producers from consumers.

Event stream: A retained ordered sequence of events that can be consumed by multiple applications.

Idempotency: The property that repeating an operation produces the same intended effect as performing it once.

Backpressure: Control applied when producers generate work faster than consumers can process it.

Circuit breaker: A mechanism that temporarily stops calls to an unhealthy dependency.

Bulkhead: Isolation of resources so that one workload cannot exhaust shared capacity.

Stateless service: A service whose durable client state is not held in a replaceable process instance.

Tail latency: High-percentile response time, such as the 99th percentile.

Graceful degradation: Providing a reduced but controlled service during partial failure.

Observability: The ability to understand internal system behavior through logs, metrics, and traces.

Review Questions

Short Answer Questions

1. Why is horizontal scaling more complex than simply adding machines?

2. What is the difference between a work queue and an event stream?

3. Why must at-least-once consumers be idempotent?

4. Distinguish liveness from readiness in health checking.

5. Give one example of an AI operation that may require strong consistency and one that may tolerate eventual consistency.

Long Answer Questions

1. Explain range partitioning, hash partitioning, and consistent hashing, including a suitable AI application for each.

2. Compare synchronous and asynchronous replication in terms of latency, availability, and data-loss risk.

3. Explain the CAP perspective and discuss why it should not be interpreted as a permanent “pick two” rule.

4. Describe a reliable event-driven AI pipeline, including retries, backpressure, schema evolution, and dead-letter processing.

5. Explain how stateless design, circuit breakers, bulkheads, and graceful degradation work together in a production inference service.

Analytical / Application Questions

1. An inference replica processes 180 requests per second. A service must sustain 1,440 requests per second after losing one replica. What is the minimum number of replicas required under this simplified model, and what practical factors could require more?

2. Design a consistency strategy for a recommendation platform containing user account status, real-time features, recommendation caches, and billing records. Justify the consistency level chosen for each.

3. A document pipeline shows rising queue age, duplicate embeddings, and occasional searches using incompatible vector indexes. Diagnose the likely distributed-systems problems and propose corrective actions.

Multiple Choice Questions with Answers

Question 1

Which partitioning strategy generally supports point lookups with relatively even distribution but makes range queries more difficult?

A. Hash partitioning

B. Range partitioning

C. Directory-free replication

D. Leader election

Correct Answer: A. Hash partitioning

Question 2

What is the principal requirement created by at-least-once message delivery?

A. Every message must be processed by exactly one consumer

B. Consumers must be idempotent or otherwise deduplicate effects

C. Messages must never be retried

D. All consumers must use synchronous replication

Correct Answer: B. Consumers must be idempotent or otherwise deduplicate effects

Question 3

During a network partition, the CAP perspective concerns a trade-off between which two properties?

A. Durability and encryption

B. Latency and throughput

C. Consistency and availability

D. Partitioning and batching

Correct Answer: C. Consistency and availability

Question 4

What is the main purpose of a circuit breaker?

A. To compress model artifacts

B. To stop calls temporarily to an unhealthy dependency

C. To guarantee exactly-once delivery

D. To partition a database by time

Correct Answer: B. To stop calls temporarily to an unhealthy dependency

Question 5

A service replica handles 250 requests per second. To sustain 3,000 requests per second after one replica fails, what is the minimum number of replicas under the stated capacity model?

A. 11

B. 12

C. 13

D. 14

Correct Answer: C. 13

Question 6

Which statement best describes eventual consistency?

A. Every read always returns the newest value

B. Replicas may temporarily differ but converge when updates stop and communication continues

C. No replica can ever fail

D. All operations require a consensus round

Correct Answer: B. Replicas may temporarily differ but converge when updates stop and communication continues

Question 7

Why is stateless service design useful for autoscaling?

A. It eliminates all persistent data

B. Any healthy instance can serve a request without relying on local durable session state

C. It guarantees zero latency

D. It prevents the use of caches

Correct Answer: B. Any healthy instance can serve a request without relying on local durable session state

Question 8

Which mechanism is most directly intended to prevent one workload from exhausting shared threads or connections?

A. Bulkhead isolation

B. Eventual consistency

C. Range partitioning

D. Model quantization

Correct Answer: A. Bulkhead isolation

Question 9

What is a key risk of unconstrained retries in a distributed service?

A. They can amplify overload and create a feedback loop

B. They always reduce availability

C. They remove the need for timeouts

D. They guarantee data freshness

Correct Answer: A. They can amplify overload and create a feedback loop

Question 10

In the illustrative document platform, why should vector indexes identify the model or preprocessing version?

A. To ensure incompatible vector spaces are not mixed unknowingly

B. To eliminate the need for queues

C. To make uploads synchronous

D. To prevent horizontal scaling

Correct Answer: A. To ensure incompatible vector spaces are not mixed unknowingly