Big Data Architectures and Ecosystems

Introduction

Big data analytics is not defined only by the size of a dataset. It also involves the speed at which data is generated, the diversity of its formats, the number of users and applications that access it, and the reliability and governance required to turn it into decisions. These conditions place demands on architecture: the organized combination of computing resources, storage systems, networks, software frameworks, data models, and operating practices used to collect, process, store, and deliver data.

Traditional business information systems commonly relied on centralized architectures. A powerful server stored the data and performed most processing. This design remains useful for modest workloads, but very large datasets and continuous data streams often exceed the storage, memory, processing, and fault-tolerance limits of one machine. Distributed architectures address this problem by dividing data and computation across multiple networked computers.

This chapter examines the major architectural ideas that support big data analytics. It compares centralized and distributed systems; explains clusters, nodes, partitioning, replication, and parallelism; introduces shared-nothing designs and the CAP theorem; surveys the Hadoop and Spark ecosystems; and evaluates batch, Lambda, Kappa, lakehouse, and data-mesh architectures. The discussion emphasizes not merely what each architecture is, but why an organization would select it, what trade-offs it creates, and how it affects analytics practice.

Learning Objectives

By the end of this chapter, you should be able to:

1. Compare centralized and distributed architectures in terms of scalability, reliability, performance, and operational complexity.
2. Explain clusters, nodes, partitions, replication, and parallel execution.
3. Describe shared-nothing systems and the role of data locality.
4. Apply the CAP theorem and distinguish common consistency models.
5. Explain the roles of Hadoop, HDFS, YARN, MapReduce, Hive, and Spark.
6. Compare batch, Lambda, Kappa, lakehouse, and data-mesh architectures.
7. Evaluate an architecture for a realistic business analytics problem, including governance, cost, and ethical considerations.

1. From Centralized Computing to Distributed Analytics

Centralized architectures

In a centralized architecture, a primary server or tightly integrated platform provides most computation and storage. Clients send requests to the central system, which manages data, executes queries, and returns results. A centralized relational database management system is a familiar example.

Centralization offers important advantages. Administration is relatively simple because data, security controls, and processing logic are concentrated. Transactions can be coordinated by one database engine, and consistency is often easier to guarantee. For many departmental applications, a well-designed centralized database is less expensive and more reliable than a distributed platform.

However, centralized systems have structural limits. A single server has finite CPU capacity, memory, storage bandwidth, and network throughput. Increasing the capacity of one machine is called vertical scaling, or scaling up. High-end hardware can be expensive, and the system may become a single point of failure unless redundant equipment is added. Even when a server is highly capable, one machine may not process a rapidly growing data volume within the required time.

Distributed architectures

A distributed architecture uses several computers connected by a network. Data and tasks are divided among them, allowing the system to scale horizontally by adding machines. A group of cooperating machines is called a cluster. Each machine is a node; a node may provide storage, computation, or both.

Distribution creates opportunities for parallelism. If a dataset is divided into partitions, different nodes can process different partitions at the same time. Distribution also supports fault tolerance through replication: maintaining multiple copies of data or intermediate results so that processing can continue after a failure.

The benefits are accompanied by costs. Network communication is slower and less predictable than communication within one computer. Nodes fail independently, clocks may differ, messages may be delayed or lost, and coordinating updates is difficult. Distributed systems therefore replace some hardware limitations with problems of coordination, consistency, monitoring, data placement, and recovery.

A distributed architecture is not automatically superior. The correct choice depends on workload size, latency requirements, transaction requirements, skills, budget, regulatory obligations, and expected growth.

2. Clusters, Nodes, Partitions, and Parallelism

Cluster organization

A cluster generally contains worker nodes and one or more coordination services. In older Hadoop terminology, a master node coordinated work while worker nodes stored and processed data. Modern systems often distribute coordination responsibilities to avoid dependence on one master. A cluster may also contain specialized nodes for ingestion, storage, query execution, stream processing, or machine learning.

A dataset is divided into partitions, sometimes called shards or blocks depending on the technology. Partitioning allows independent processing and improves scalability. A good partitioning strategy distributes work evenly. If one partition is much larger or more frequently accessed than others, a data-skew problem occurs. The overloaded partition becomes a bottleneck while other nodes remain underused.

Common partitioning methods include:

• Hash partitioning: a hash function maps a key to a partition. It usually distributes records well but does not preserve ordering.
• Range partitioning: records are assigned according to value ranges, such as dates or customer identifiers. It supports range queries but may create imbalance when values are unevenly distributed.
• Round-robin partitioning: records are assigned sequentially across partitions. It balances record counts but may not colocate related records.
• Composite partitioning: two or more methods are combined, for example partitioning by date and then hashing by customer.

Partitioning is both a performance decision and a governance decision. A customer identifier may support efficient customer-level analysis, but it can also increase the concentration of sensitive information in particular locations. Access control and encryption must therefore operate at the appropriate data and partition levels.

Parallelism

Parallelism means performing multiple computational activities concurrently. Data parallelism applies the same operation to separate data partitions. Task parallelism executes different operations at the same time. Pipeline parallelism overlaps stages, such as reading, transforming, and writing data.

A useful upper-bound model is Amdahl’s law. If a fraction p of a workload can be parallelized and the remaining fraction 1 − p must execute serially, the theoretical speedup with N parallel workers is:

Here, S(N) is speedup relative to one worker, p is the parallelizable fraction, and N is the number of workers. The formula assumes that worker performance is comparable and ignores communication, scheduling, input/output, and fault-recovery overhead. It therefore gives an upper bound rather than a guaranteed performance result.

For example, if p = 0.90 and N = 10:

Although ten workers are available, the workload can be at most about 5.26 times faster under these assumptions because 10 percent remains serial. In a real cluster, network and coordination costs may reduce the observed speedup further.

Data locality and shuffles

Data locality is the principle of moving computation toward data rather than moving large datasets across the network. It was central to early Hadoop design. If a node already stores a block, a task can process it locally, reducing network traffic.

Some operations require a shuffle: records are redistributed between nodes so that related records meet. A distributed join, grouping, or sorting operation commonly causes a shuffle. Shuffles are expensive because they use network bandwidth, disk, serialization, and memory. Analysts and engineers improve performance by filtering early, selecting only required columns, choosing suitable partition keys, using broadcast joins for genuinely small tables, and avoiding unnecessary repartitioning.

3. Shared-Nothing Architecture

A shared-nothing system assigns each node its own processor, memory, and local storage. Nodes do not depend on a common disk or common memory pool for normal operation. Communication takes place over the network, and data is partitioned among nodes.

Shared-nothing design has three major advantages. First, it scales horizontally: additional nodes add processing and storage capacity. Second, failure isolation is possible because one node can fail without making every other node unavailable. Third, workloads can be distributed according to data placement. These properties explain its importance in massively parallel processing databases and big data frameworks.

The design also introduces challenges. A query involving records stored on many nodes requires network coordination. A poor partition key can create hotspots or expensive cross-node joins. Rebalancing data when nodes are added or removed can be disruptive. Replication consumes storage and creates the need to reconcile multiple copies.

Shared-disk systems take a different approach: many compute nodes access a common storage system. They can simplify data sharing and may support flexible scaling of compute, but the shared storage layer can become a performance or availability dependency. Cloud data warehouses and object-storage-based analytics platforms often combine separated compute and storage rather than fitting perfectly into either traditional category.

4. Fault Tolerance, Replication, and Recovery

Failures are normal in a large cluster. A disk may fail, a process may crash, a network switch may become unavailable, or a software defect may produce incorrect output. A reliable architecture assumes failure rather than treating it as an exceptional event.

Replication stores multiple copies of data. A replication factor of three means that, conceptually, three copies are maintained, although implementation details vary. Replication improves availability and read performance but increases storage cost and creates consistency obligations.

Redundancy may also be achieved through erasure coding, which stores data and parity information. Erasure coding can use less storage than full replication for durable archival data, but reconstruction is computationally and operationally more complex.

Checkpointing saves intermediate state so that a failed computation can resume from a known point. Recomputation regenerates lost results from the original data and transformation logic. Spark, for example, can often reconstruct lost partitions from a lineage of transformations. The appropriate method depends on the cost of recomputation, the reliability of source data, and the time required to recover.

Reliability should be measured against explicit service objectives. Availability is often expressed as the proportion of time a service is operational. If a service is available for A hours during a measurement period of T hours, its observed availability is:

The formula is a ratio with identical time units in the numerator and denominator. A target such as 99.9 percent is a service objective, not a guarantee that every query will succeed or that data will always be correct.

5. CAP Theorem and Consistency Models

CAP theorem

The CAP theorem states that in the presence of a network partition, a distributed data system cannot simultaneously guarantee both strong consistency and availability for every request. The three properties are:

• Consistency: every successful read receives the most recent successful write, according to the system’s defined ordering.
• Availability: every request to a non-failing node receives a response, although that response may not contain the newest value.
• Partition tolerance: the system continues operating despite a communication breakdown between nodes.

Network partitions cannot be completely avoided in a distributed system. Consequently, a system must make a trade-off during a partition: it may reject or delay some requests to preserve consistency, or it may respond with potentially stale data to preserve availability. CAP does not mean that a system permanently chooses only two properties, nor does it say that consistency and availability have no meaning outside a partition. It describes a fundamental trade-off under partition conditions.

For example, a banking transfer may require consistency: accepting conflicting account balances is unacceptable. A social-media recommendation feed may favor availability and tolerate temporarily stale results. The choice depends on business risk, not merely on technical preference.

Consistency models

Consistency is not a single universal setting. Common models include:

• Strong or linearizable consistency: an operation appears to take effect at one instant between its invocation and completion, and later reads observe it. This model is easier for users to reason about but may require coordination and reduce availability during failures.
• Sequential consistency: operations appear in one total order consistent with the order of operations issued by each individual process. It is weaker than linearizability because real-time ordering across processes is not necessarily preserved.
• Causal consistency: causally related operations are observed in the same order by all users, while concurrent unrelated operations may be seen in different orders.
• Eventual consistency: if no new updates occur and communication continues, replicas eventually converge. Reads may temporarily return stale values.
• Read-your-writes consistency: a user’s later reads reflect that user’s completed writes.
• Monotonic reads: once a user has observed a value, subsequent reads do not return an older version.

These models are important in analytics. A daily financial report may use a carefully defined snapshot and therefore avoid exposing partially loaded data. A real-time dashboard may accept a small delay in exchange for responsiveness. Data contracts should state freshness and consistency expectations explicitly.

6. The Hadoop Ecosystem

Apache Hadoop established a widely influential model for large-scale distributed storage and batch processing. Its principal components include:

• Hadoop Distributed File System (HDFS): a distributed file system that stores large files as blocks across nodes and replicates blocks for fault tolerance. It is optimized for high-throughput access to large files rather than low-latency individual record updates.
• YARN: a resource-management and job-scheduling layer. It separates cluster resource management from application execution.
• MapReduce: a programming model in which map tasks transform input records and reduce tasks aggregate or combine intermediate results. The framework manages partitioning, scheduling, and recovery.
• Hive: a data-warehouse interface that supports SQL-like queries over large datasets. Hive translates queries into execution plans rather than requiring every analyst to write Java MapReduce code.
• HBase: a distributed wide-column database designed for random, near-real-time access to very large tables. It is conceptually different from HDFS, which is primarily a file system.
• Sqoop and Flume: historically used for relational database transfer and event/log ingestion, respectively. Their relevance depends on the specific version and deployment environment.

Hadoop’s strengths include economical scale-out storage, fault tolerance, and strong support for large batch workloads. Its weaknesses include operational complexity, relatively high latency for interactive analysis, and the need to manage many services. Modern cloud systems and newer table formats have replaced or complemented some traditional Hadoop deployments, but Hadoop concepts remain foundational.

7. Apache Spark and Modern Processing

Apache Spark is a distributed processing engine designed for batch, interactive, streaming, SQL, and machine-learning workloads. It uses a driver program to coordinate execution and executor processes to perform tasks. Spark applications construct logical operations on distributed data, and the engine creates physical execution plans.

Spark’s DataFrame and Dataset APIs provide structured processing, while Spark SQL supports relational queries. Structured Streaming extends the same programming model to continuously arriving data. Spark can run with cluster managers such as standalone mode, YARN, or Kubernetes, and it can read from systems including object stores, HDFS, relational databases, and message brokers.

Spark transformations are generally lazy: the engine records a computation plan and executes it when an action requires a result. This permits optimization, such as predicate pushdown and column pruning. Narrow transformations, such as filtering, can often be performed without redistributing data. Wide transformations, such as grouping by key, commonly require a shuffle.

Spark is not a storage system in the same sense as HDFS or an object store. It is a compute engine that relies on durable storage for input and output. Nor is Spark automatically a real-time system: streaming latency depends on source, trigger, state management, cluster resources, and workload design.

A practical Hadoop-and-Spark architecture may therefore use object storage or HDFS for durable data, a message broker for events, Spark for transformation and machine learning, a table format for reliable analytical storage, and a SQL warehouse or dashboard platform for consumption.

8. Major Big Data Architectural Patterns

Batch architecture

A batch architecture collects data and processes it periodically. For example, sales transactions may be extracted overnight, transformed, validated, and loaded into an analytical warehouse each morning. Batch processing is efficient for large scans and complex transformations, and it is relatively easy to make a run reproducible.

Its limitation is latency. Decisions cannot use information that has not yet been processed. Batch jobs can also create a large recovery window when a failure occurs near the end of a scheduled run. Batch remains appropriate when hourly, daily, or weekly freshness is sufficient.

Lambda architecture

Lambda architecture combines a batch layer, a speed layer, and a serving layer. The batch layer computes accurate views from the complete historical dataset. The speed layer processes recent events rapidly to reduce latency. The serving layer combines or exposes the results.

The principal advantage is the ability to provide both historical correctness and low-latency updates. The principal disadvantage is duplicated logic: developers often implement similar business transformations in both batch and speed paths. Differences between the paths can produce inconsistent results, and operations become more complex.

Kappa architecture

Kappa architecture uses a streaming-first design. Events are retained in an ordered, replayable log, and one stream-processing path computes current views. When logic changes, the system replays historical events through the new application.

Kappa reduces duplicate processing paths, but it requires durable event retention, deterministic processing, effective state management, and a stream engine capable of replaying substantial history. It is especially suitable when the business can represent changes as events and when near-real-time processing is central.

Lakehouse architecture

A lakehouse combines the low-cost, flexible storage of a data lake with management capabilities traditionally associated with a warehouse. A lakehouse commonly uses object storage, open file formats such as Parquet, and a transactional table layer that provides schema management, versioning, partition evolution, and reliable updates.

The architectural goal is to reduce copying between separate lakes and warehouses while supporting BI, data science, machine learning, and streaming. A lakehouse does not automatically solve data quality or governance problems. Without ownership, cataloging, access controls, and disciplined table design, it can become another data swamp.

Data-mesh architecture

Data mesh is an organizational and architectural approach based on four principles: domain-oriented ownership, data as a product, self-serve data infrastructure, and federated computational governance. A sales domain, for example, may own a well-documented sales data product rather than sending every request to a central data team.

Data mesh addresses organizational bottlenecks, not merely technical scale. It can improve accountability and domain knowledge, but it requires strong standards for identity, metadata, quality, interoperability, privacy, and platform operations. Without governance and investment in self-service infrastructure, decentralization can create fragmented and incompatible data products.

9. Selecting an Architecture

Architecture selection should begin with business requirements rather than fashionable technologies. Important questions include:

1. How fresh must the data be: seconds, minutes, hours, or days?
2. Is the workload primarily transactional, analytical, operational, or mixed?
3. What volume and velocity are expected over the next three to five years?
4. Are updates, deletes, and point lookups required, or are large scans dominant?
5. What consistency level is necessary for each data product?
6. What are the recovery-point and recovery-time objectives?
7. Which regulations apply to personal, financial, health, or location data?
8. Does the organization possess the engineering and governance capability to operate the design?
9. Can the architecture be tested, monitored, audited, and retired safely?

A useful design separates concerns without creating unnecessary copies. Durable raw data may be retained for reproducibility, curated tables may support reliable analytics, and serving stores may be optimized for dashboards or applications. Each layer should have an owner, a quality definition, a retention policy, and documented access rules.

10. Illustrative Case: A National Retailer’s Inventory Analytics

Background

Consider an illustrative retailer with 1,200 stores, an online shop, warehouse systems, supplier feeds, and millions of daily product and customer events. Store sales are recorded in local systems, online orders arrive continuously, and inventory decisions must account for deliveries, returns, cancellations, and stock adjustments.

Problem

The retailer’s centralized reporting database receives nightly extracts. Managers can view yesterday’s sales, but online stock availability is often several hours out of date. During promotions, the reporting server becomes overloaded. Different departments also calculate “available inventory” differently, producing conflicting dashboards.

Analysis

The workload contains at least three distinct requirements. First, historical sales and inventory analysis involves large scans and can be processed in batch. Second, online availability requires low-latency updates. Third, inventory figures need clearly defined semantics: a product reserved for an unshipped order should not be reported as freely available.

A single centralized database could be upgraded, but scaling the server would not fully solve event ingestion, fault isolation, or conflicting definitions. A distributed design is more appropriate, provided that the organization accepts the cost of operating it.

Approach and solution

The retailer adopts an event-oriented lakehouse design with the following components:

• Store, warehouse, and online systems publish sales, receipt, return, reservation, and adjustment events to a durable message platform.
• A streaming process validates event structure, assigns event time, removes exact duplicates using event identifiers, and creates a near-real-time inventory view.
• Raw events are retained in object storage for audit and replay.
• Curated transactional tables maintain standardized product, location, and inventory facts.
• Batch jobs reconcile late-arriving events, correct historical records, and calculate daily performance measures.
• A governed semantic layer defines measures such as on-hand, reserved, damaged, and available-to-promise inventory.
• Dashboards use a serving layer optimized for queries rather than directly scanning all raw events.

The architecture uses an eventual-consistency model for some dashboards: a view may be seconds behind the latest event. However, the order-fulfillment service uses a stronger consistency mechanism for reservations so that two customers are not promised the same scarce item. This illustrates why consistency should be selected per business operation rather than imposed uniformly on every dataset.

Outcome

The retailer reduces reporting delay from one day to several minutes for operational views and preserves a historical record for reconciliation. During a node failure, replicated event data and replayable processing allow recovery. The design does not eliminate errors: late events, incorrect supplier data, and ambiguous product identities remain governance problems. The organization therefore creates data-product ownership, automated quality checks, lineage documentation, and an incident process.

Learning points

The case demonstrates that architecture is a portfolio of decisions. Batch remains valuable for reconciliation and historical computation; streaming supports operational freshness; durable storage supports replay; and semantic governance ensures that technically available data has a consistent business meaning.

11. Governance, Security, and Ethical Responsibilities

Distributed architectures expand the number of systems, copies, identities, and vendors that must be governed. Security should include encryption in transit and at rest, least-privilege access, network segmentation, secrets management, strong authentication, and auditable administrative actions.

Data minimization is especially important. An organization should collect and retain only data justified by a legitimate purpose. Sensitive attributes may require masking, tokenization, aggregation, or controlled access. Replication and backups can unintentionally create additional copies of personal data, so retention and deletion policies must cover every layer, including caches and derived tables.

Lineage records where data originated, how it was transformed, and which reports depend on it. Quality controls should test completeness, validity, uniqueness, timeliness, and consistency. A technically available dataset is not necessarily fit for a high-impact decision.

Analytics architectures also influence fairness and accountability. If a predictive model uses stale or unevenly collected data, its outputs may disadvantage particular groups. Streaming systems can make decisions rapidly while reducing opportunities for human review. Organizations should document purposes, assess risks, monitor outcomes, provide appropriate explanations, and comply with applicable privacy, records-management, and sector-specific regulations.

12. Contemporary Developments and Best Practices

Current practice increasingly separates storage and compute so that each can scale independently. Open table formats, columnar storage, metadata catalogs, workload orchestration, and declarative transformation tools are reducing dependence on proprietary processing paths. Container orchestration and managed cloud services can simplify deployment, but they do not remove the need for capacity management, security, cost controls, or architecture expertise.

Effective practice includes the following:

• Design around measurable service-level objectives for freshness, availability, latency, and recovery.
• Choose partition keys using access patterns, cardinality, growth, and skew analysis.
• Minimize expensive shuffles and unnecessary copies.
• Treat schemas, data contracts, and metric definitions as governed assets.
• Make pipelines idempotent where possible, so rerunning a job does not create incorrect duplicates.
• Retain sufficient raw evidence for audit and replay, subject to lawful retention limits.
• Monitor both infrastructure and data: CPU, memory, network, lag, null rates, duplicate rates, and freshness.
• Test failure recovery, not only successful execution.
• Select consistency models according to business consequences.
• Build cost accountability into design through workload budgets, lifecycle policies, and resource tagging.
• Prefer the simplest architecture that satisfies verified requirements.

Chapter Summary

Big data architectures distribute storage and computation across clusters because one machine may not provide sufficient capacity, throughput, or resilience. Nodes store and process partitions, while parallelism enables multiple tasks to execute concurrently. Amdahl’s law shows why serial work limits speedup even when more workers are added. Shared-nothing systems scale by giving nodes independent resources, but they require careful partitioning, shuffle management, replication, and recovery design.

The CAP theorem explains that, during a network partition, a distributed system must trade off strong consistency and availability. Consistency models range from linearizable to eventual, and the appropriate choice depends on business risk. Hadoop introduced durable distributed storage and batch processing through HDFS, YARN, and MapReduce; Spark provides a flexible distributed compute engine for SQL, batch, streaming, and machine learning.

Batch, Lambda, Kappa, lakehouse, and data-mesh architectures address different technical and organizational needs. Batch favors efficiency and reproducibility; Lambda combines historical and low-latency paths; Kappa emphasizes replayable streams; lakehouse designs unite flexible storage with managed analytical tables; and data mesh distributes ownership through governed data products. Architecture selection must consider latency, scale, consistency, reliability, skills, cost, security, privacy, and data governance.

Key Terms

Availability: The ability of a system to respond to requests when required.

Amdahl’s law: A model showing how the serial portion of a workload limits parallel speedup.

Batch processing: Processing accumulated data in scheduled groups.

CAP theorem: The result that, during a network partition, strong consistency and full availability cannot both be guaranteed for every request.

Cluster: A group of networked computers cooperating as one computing system.

Consistency model: A formal description of what values reads may observe after writes occur.

Data locality: Processing data near the node where it is stored.

Data mesh: An organizational approach based on domain ownership, data products, self-service infrastructure, and federated governance.

Data partition: A subset of a dataset assigned to a node or processing unit.

Eventual consistency: A model in which replicas converge if updates stop and communication continues.

HDFS: Hadoop’s distributed file system for storing large files across nodes.

Kappa architecture: A streaming-first architecture that uses replayable events as the basis for processing.

Lakehouse: An analytical architecture combining data-lake flexibility with warehouse-like management features.

Lambda architecture: An architecture combining batch and speed-processing paths with a serving layer.

Node: An individual computer or execution instance in a cluster.

Parallelism: Concurrent execution of multiple computational activities.

Replication: Maintaining multiple copies of data or state.

Shared-nothing architecture: A design in which nodes have independent processors, memory, and storage.

Shuffle: Redistribution of data among nodes during distributed processing.

YARN: Hadoop’s resource-management and scheduling layer.

Review Questions

Short Answer Questions

1. What is the main difference between vertical and horizontal scaling?
2. Why can data skew reduce the performance of a distributed job?
3. What does replication provide, and what cost does it create?
4. Define eventual consistency.
5. What is the role of Spark in a big data ecosystem?

Long Answer Questions

1. Compare centralized and distributed architectures with respect to scalability, fault tolerance, consistency, and operational complexity.
2. Explain shared-nothing architecture, including partitioning, data locality, shuffles, and rebalancing challenges.
3. State and explain the CAP theorem, then distinguish linearizable, causal, and eventual consistency.
4. Compare Hadoop and Spark in terms of storage, computation, execution style, and typical analytical workloads.
5. Evaluate batch, Lambda, Kappa, lakehouse, and data-mesh architectures, identifying an appropriate use case and limitation for each.

Analytical / Application Questions

1. A workload is 80 percent parallelizable. Using Amdahl’s law, calculate the theoretical speedup with eight workers and explain why the result is lower than eight times.
2. Design an architecture for a food-delivery platform that needs historical profitability reports and near-real-time driver and order tracking. Justify the processing, storage, consistency, and governance choices.
3. An organization proposes a data mesh but has no common identity system, metadata catalog, or data-quality standards. Analyze the risks and recommend a staged implementation plan.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly distinguishes horizontal scaling from vertical scaling?

A. Horizontal scaling adds or removes machines; vertical scaling increases the capacity of one machine.

B. Horizontal scaling always provides strong consistency; vertical scaling never does.

C. Horizontal scaling uses only batch processing; vertical scaling uses only streaming.

D. Horizontal scaling eliminates the need for network communication.

Correct Answer: A. Horizontal scaling adds or removes machines; vertical scaling increases the capacity of one machine.

Question 2

What is a likely consequence of severe data skew in a distributed job?

A. All partitions finish simultaneously because records are evenly distributed.

B. One or more nodes become bottlenecks while other nodes are underused.

C. Replication is automatically disabled.

D. The dataset becomes strongly consistent.

Correct Answer: B. One or more nodes become bottlenecks while other nodes are underused.

Question 3

Using Amdahl’s law, what is the theoretical speedup when p = 0.80 and N = 8?

A. 2.50

B. 3.33

C. 4.00

D. 6.40

Correct Answer: B. 3.33

Question 4

According to the CAP theorem, what trade-off arises during a network partition?

A. A system must trade storage for computation.

B. A system cannot guarantee both strong consistency and availability for every request.

C. A system must stop using replication permanently.

D. A system cannot perform any batch processing.

Correct Answer: B. A system cannot guarantee both strong consistency and availability for every request.

Question 5

Which consistency model allows replicas to return temporarily stale data but requires convergence if updates stop and communication continues?

A. Linearizable consistency

B. Sequential consistency

C. Eventual consistency

D. Causal consistency

Correct Answer: C. Eventual consistency

Question 6

What is the primary role of HDFS in the Hadoop ecosystem?

A. Managing user passwords

B. Providing distributed storage for large files

C. Serving interactive dashboard visualizations

D. Training only neural-network models

Correct Answer: B. Providing distributed storage for large files

Question 7

Why are wide transformations in Spark often expensive?

A. They generally require a shuffle of data between nodes.

B. They cannot be executed on more than one node.

C. They permanently disable lazy evaluation.

D. They require all data to fit in one executor’s memory.

Correct Answer: A. They generally require a shuffle of data between nodes.

Question 8

Which architecture combines a batch layer and a speed layer?

A. Kappa architecture

B. Lambda architecture

C. Data-mesh architecture

D. Shared-nothing architecture

Correct Answer: B. Lambda architecture

Question 9

Which statement best describes a lakehouse?

A. A system used exclusively for low-latency transactions

B. A design combining flexible lake storage with warehouse-like management capabilities

C. A network protocol for node discovery

D. A replacement for all data governance processes

Correct Answer: B. A design combining flexible lake storage with warehouse-like management capabilities

Question 10

Which principle is central to data mesh?

A. All data must be owned by one central database administrator.

B. Every dataset must be processed only once per year.

C. Business domains own governed data products supported by self-service infrastructure.

D. Data quality is unnecessary when storage is replicated.

Correct Answer: C. Business domains own governed data products supported by self-service infrastructure.