Big Data Architecture and Design

Introduction

Big data engineering is not simply the selection of a database or processing framework. It is the disciplined design of a socio-technical system that acquires, stores, processes, governs, and delivers data at appropriate levels of scale, speed, reliability, and cost. Architecture provides the enduring structure of that system; design translates business and technical requirements into deployable components, interfaces, controls, and operating practices.

The architectural choices considered in this chapter—data lakes, warehouses, lakehouses, data meshes, data fabrics, and Lambda and Kappa processing patterns—address different problems. They are not interchangeable labels. A data lake primarily describes a storage and data-management approach; a warehouse emphasizes curated analytical data; a lakehouse combines low-cost, flexible storage with warehouse-like management; a data mesh emphasizes organizational ownership and domain-oriented data products; a data fabric emphasizes integrated metadata, interoperability, and automation; and Lambda and Kappa describe patterns for handling batch and streaming computation.

A sound architecture begins with requirements rather than fashion. The designer must determine whether data is structured, semi-structured, or unstructured; whether workloads are analytical, operational, or both; how quickly results are required; who owns data quality; what regulatory constraints apply; and how much operational complexity the organization can sustain.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the roles and trade-offs of lakes, warehouses, lakehouses, meshes, and fabrics.
2. Select centralized or decentralized ownership models according to organizational and data requirements.
3. Compare batch, streaming, Lambda, and Kappa processing patterns.
4. Design an architecture using explicit requirements, quality attributes, and data contracts.
5. Evaluate security, governance, cost, reliability, and ethical risks in big data systems.
6. Justify architectural decisions in a realistic organizational case.

1. Architectural Foundations

1.1 Architecture as a set of decisions

A big data architecture specifies at least five dimensions:

• Data sources: applications, devices, logs, partners, sensors, files, and public data.
• Ingestion: batch transfer, message brokers, change-data capture, application programming interfaces, and file exchange.
• Storage: object stores, distributed file systems, analytical databases, operational databases, and archival systems.
• Processing and serving: batch engines, stream processors, SQL query engines, feature stores, dashboards, APIs, and machine-learning platforms.
• Governance and operations: identity, lineage, metadata, quality, retention, observability, backup, disaster recovery, and cost management.

Architecture is evaluated through quality attributes rather than component count. Important attributes include scalability, availability, latency, consistency, durability, security, privacy, maintainability, portability, and total cost of ownership. These attributes conflict. For example, stronger consistency can increase latency or reduce availability during network partitions; extensive replication can improve resilience while increasing storage cost.

1.2 Workload dimensions

A useful design begins by classifying workloads.

Two quantitative measures are especially useful. If a system processes a volume of data V in time T, its average throughput is:

Here R is throughput, measured in bytes per second or records per second; V is processed volume; and T is elapsed processing time. This is an average, not a guarantee of peak capacity. Design capacity should include peak arrival rates, burstiness, processing overhead, and failure recovery.

For an event system, end-to-end latency can be decomposed approximately as:

Each term represents a non-negative latency component. The approximation is useful for identifying bottlenecks, assuming the stages are sequential and measured consistently. Parallel processing, batching, retries, and queueing can make actual latency distributions more complex; therefore, architects should examine percentiles such as p95 and p99 rather than only averages.

2. Major Data Architecture Models

2.1 Data warehouses

A data warehouse is a managed repository optimized for structured, integrated, and analytical data. Traditional warehouses commonly use dimensional models consisting of fact tables and dimension tables. Modern cloud warehouses may support semi-structured data, elastic compute, columnar storage, and separation of storage from compute.

Warehouses are strong when an organization needs governed metrics, predictable SQL performance, conformed dimensions, controlled access, and reliable reporting. They are less suitable as the sole repository for raw multimedia, high-volume machine data, or rapidly changing exploratory data. Transformation commonly occurs before data is published to analytical users, although contemporary approaches may load raw data first and transform inside the warehouse.

A warehouse should not be confused with an operational database. An operational database is optimized for transaction processing, concurrency, and low-latency updates; a warehouse is optimized for scans, aggregations, historical analysis, and workload isolation.

2.2 Data lakes

A data lake stores large quantities of raw or lightly processed data, often in object storage using open file formats. It can accommodate structured, semi-structured, and unstructured data. Schema-on-read permits different consumers to interpret data differently, which supports experimentation and delayed modeling.

The flexibility of a lake creates risks. Without cataloguing, quality rules, ownership, retention policies, and controlled publication, a lake can become a “data swamp”: data may be duplicated, undocumented, inaccessible, or impossible to trust. A production lake therefore needs zones or equivalent lifecycle controls, such as:

• Landing or raw: immutable copies of source data, with ingestion metadata.
• Standardized or cleansed: validated types, normalized formats, and quality flags.
• Curated or serving: domain-specific datasets with documented semantics and access policies.

These zones are logical controls rather than mandatory physical directories. The central principle is that raw data should remain traceable while derived data should be reproducible.

2.3 Lakehouses

A lakehouse combines inexpensive, scalable data-lake storage with table management, transactions, schema enforcement, versioning, and analytical performance features traditionally associated with warehouses. Its purpose is to reduce duplication between a lake used by engineers and a warehouse used by analysts.

A lakehouse is attractive when one organization needs data science, business intelligence, and large-scale engineering on a common foundation. It may support open table formats, columnar files, partition pruning, compaction, time travel, and transactional updates. However, the term does not eliminate design problems. Transaction guarantees may apply only within a table or platform; streaming and batch workloads still require careful coordination; and specialized low-latency or highly concurrent workloads may need separate serving systems.

2.4 Data mesh

Data mesh is primarily an organizational and architectural paradigm. Its four widely cited principles are:

1. Domain ownership: teams close to the business domain own data.
2. Data as a product: datasets have users, quality expectations, documentation, service levels, and support.
3. Self-serve data platform: a platform team provides reusable infrastructure and guardrails.
4. Federated computational governance: governance is shared across domains but implemented through common policies and automation.

A mesh is not achieved merely by assigning each department a database. Domain teams need capability, incentives, product-management practices, and interoperability standards. A mesh can improve responsiveness and contextual quality, but it can also produce inconsistent definitions, duplicated platforms, and unequal maturity across domains.

2.5 Data fabric

Data fabric is an architectural approach centered on connecting heterogeneous data environments through metadata, lineage, semantic models, integration services, and policy-driven automation. It is particularly relevant where data is distributed across clouds, data centers, applications, and domains.

A fabric may use active metadata to discover assets, recommend relationships, automate quality checks, apply access rules, and expose data through consistent interfaces. Unlike a mesh, which foregrounds ownership and organizational structure, a fabric foregrounds integration and intelligence across environments. The two can coexist: a mesh may define who owns a data product, while a fabric helps users discover, govern, and connect it.

2.6 Comparative evaluation

3. Centralized, Decentralized, and Hybrid Ownership

3.1 Centralized ownership

In a centralized model, a data or analytics unit controls ingestion, modeling, governance, and often platform operations. This supports consistent standards, common security controls, and economies of scale. It is effective when data domains are closely related, the organization is small, or regulatory control is paramount.

Its weaknesses include a queue of requests, limited domain understanding, and the risk that business teams treat data as an externally provided service. Central teams may optimize for platform stability while missing changing domain needs.

3.2 Decentralized ownership

In a decentralized model, domain teams own and operate their data products. Ownership is closer to the source, and teams can respond quickly to local requirements. The model is suitable for large organizations with distinct business domains and strong engineering maturity.

Decentralization requires common standards for identity, metadata, quality, interoperability, retention, and incident management. Without them, the organization experiences semantic divergence—for example, different definitions of “active customer”—and duplicated pipelines.

3.3 Hybrid governance

Most institutions benefit from a hybrid model. Domain teams own data products and quality at the source, while a central platform group provides infrastructure, reference patterns, identity services, observability, and automated controls. A governance council defines enterprise-wide policies, but implementation should be embedded in development workflows rather than left to documents.

Ownership should be explicit. A useful responsibility matrix assigns each dataset a business owner, technical owner, steward, custodian, and approved consumer group. The business owner decides meaning and acceptable use; the technical owner operates pipelines and interfaces; the steward manages definitions and quality; the custodian manages infrastructure and security.

4. Batch, Streaming, Lambda, and Kappa Architectures

4.1 Batch processing

Batch processing collects data and processes it periodically. It is economical and operationally straightforward for payroll, monthly reporting, historical aggregation, and workloads where minutes or hours of delay are acceptable. Batch systems are also valuable for backfills and authoritative recomputation.

Batch design must address late-arriving data, duplicate files, partial failures, reruns, partitioning, and reconciliation. Idempotence is important: executing the same logical job more than once should not produce unintended duplicate effects. This can be achieved through deterministic transformations, unique event identifiers, controlled overwrite operations, and checkpointed progress.

4.2 Stream processing

Stream processing handles continuously arriving events. It is appropriate for fraud detection, equipment monitoring, personalization, operational alerting, and telemetry. A stream architecture must specify event-time semantics, ordering assumptions, delivery guarantees, late-event handling, state management, replay, and retention.

“Exactly once” is often used imprecisely. It may refer to processing semantics, state updates, or externally visible effects. End-to-end exactly-once behavior requires coordination across the source, processor, state store, and sink; otherwise, systems commonly provide at-least-once delivery with deduplication or idempotent writes.

4.3 Lambda architecture

Lambda architecture uses two paths:

• A batch layer stores the complete dataset and periodically computes accurate views.
• A speed layer processes recent events rapidly to provide low-latency results.
• A serving layer combines or exposes the batch and speed results.

Its major advantage is resilience: the batch layer can correct errors and rebuild views. Its principal cost is maintaining two processing implementations, which increases code duplication, semantic inconsistency, testing effort, and operational burden.

4.4 Kappa architecture

Kappa architecture uses a stream-processing path as the primary computation model. Historical data is retained as an immutable event log or replayable source, allowing a revised application to recompute results by replaying events. Kappa is attractive when events are naturally ordered, replay is practical, and streaming tools can handle both recent and historical workloads.

Kappa does not mean that batch jobs disappear. Backfills, model training, archival, and administrative tasks may remain batch-oriented. The distinction is that the principal data-processing logic is unified around the stream.

4.5 Choosing a processing pattern

Architects should avoid choosing Lambda or Kappa as an ideological decision. The decision depends on freshness objectives, replayability, event quality, team skills, stateful processing needs, and the cost of operating duplicate paths.

Figure: Lambda maintains separate batch and speed paths, whereas Kappa uses a replayable stream-processing path.

5. A Method for Architectural Design

5.1 Establish requirements and service levels

Requirements should be measurable. Instead of stating “real time,” specify a target such as “95% of valid events visible in the dashboard within 30 seconds.” Define availability, recovery-point objective (maximum acceptable data loss measured in time), recovery-time objective (maximum acceptable restoration time), retention, data quality thresholds, and expected peak volume.

A simple capacity estimate can be made when arrival rate is approximately stable. If r is the average arrival rate in records per second and s is average record size in bytes, the average ingress bandwidth is:

The estimate assumes a representative average size and excludes protocol overhead, replication, indexing, and compression. Capacity planning must add those factors and account for peak rate rather than relying only on the average.

5.2 Define data products and contracts

A data contract states the producer’s schema, semantic definitions, quality expectations, ownership, compatibility rules, and change-notification process. Contracts reduce accidental breakage between producers and consumers. They should address field types, nullability, units, timestamps, identifiers, enumerations, privacy classification, and versioning.

A data product should have a clear purpose, documentation, discoverability, access mechanism, freshness target, quality measurements, lineage, and support contact. Publishing a table without these properties is not equivalent to delivering a product.

5.3 Select storage and processing components

Select technologies according to workload rather than brand. Object storage is often appropriate for durable, large-scale files; a warehouse is appropriate for governed SQL analytics; a lakehouse may unify engineering and analytical workloads; a key-value or search system may serve low-latency lookups; and a time-series store may suit telemetry. Polyglot persistence is acceptable when boundaries and operational ownership are explicit.

Partitioning should support common filters, avoid excessive small files, and account for skew. Compression and columnar formats reduce storage and scan costs, but overly aggressive partitioning can increase metadata overhead. Data layout should be measured through query plans and operational metrics rather than assumed from naming conventions.

5.4 Build governance into the platform

Important controls include:

• identity-based access and least privilege;
• encryption in transit and at rest;
• cataloguing, lineage, and classification;
• row-, column-, or record-level restrictions where necessary;
• quality checks for completeness, validity, uniqueness, timeliness, and consistency;
• retention and deletion workflows;
• immutable audit logs;
• backup, restore, and disaster-recovery tests.

Privacy engineering should apply data minimization, purpose limitation, pseudonymization or anonymization where appropriate, and controlled re-identification. Regulations vary by jurisdiction, but legal compliance is not a substitute for ethical judgment. Data that can technically be collected may still be inappropriate to use.

5.5 Operate and observe

Operational maturity requires monitoring of pipeline success, lag, throughput, freshness, schema changes, data quality, cost, access anomalies, and infrastructure health. Alerts should be tied to user impact and service objectives rather than every technical fluctuation. Incident processes should preserve evidence, communicate clearly, and include post-incident learning.

6. Worked Design Example

An online retailer receives orders, clickstream events, inventory updates, and supplier files. Fraud decisions must be produced within 10 seconds, executive sales dashboards may be refreshed every 15 minutes, and financial reconciliation occurs nightly. Historical data must support machine-learning experiments and audited reporting.

A single traditional warehouse would provide good reporting but would be inefficient for raw clickstream and immediate fraud decisions. A pure decentralized mesh could preserve domain autonomy but might create inconsistent customer and revenue definitions. A suitable design is hybrid:

1. Events enter a durable log through validated schemas and unique event identifiers.
2. A streaming processor computes fraud features and alerts, using event time and a controlled lateness policy.
3. Raw events are retained in a lakehouse in immutable, access-controlled form.
4. Curated order, customer, and product data are published as governed data products.
5. A warehouse or warehouse-compatible serving layer provides conformed sales metrics every 15 minutes.
6. Nightly batch reconciliation compares order, payment, and shipment records and publishes authoritative financial tables.
7. A central platform team supplies identity, cataloguing, lineage, deployment templates, and observability; domain teams own order, fraud, inventory, and supplier products.

The design uses streaming for fraud, micro-batch or incremental processing for dashboards, and batch for reconciliation. It does not require Lambda if the streaming and historical computations can share replayable event-based logic. If financial corrections rely heavily on external files and complex restatements, a separate batch reconciliation process remains justified.

The principal risks are duplicate events, late inventory updates, definition conflicts, excessive storage of personal data, and untested recovery procedures. Controls include idempotent event handling, reconciliation reports, data contracts, customer-data minimization, role-based access, and regular restoration exercises.

7. Critical Evaluation and Contemporary Developments

7.1 Cost and complexity

Cloud elasticity does not guarantee low cost. Compute may scale automatically while inefficient queries, repeated transformations, data egress, duplicate copies, and excessive retention accumulate expense. FinOps practices should connect cost to products and workloads, establish budgets, identify idle resources, and evaluate the value of freshness and precision.

7.2 Open standards and portability

Open table formats, open columnar formats, portable SQL, and standard event schemas can reduce lock-in. Portability is not free: different engines implement features differently, and migration requires testing semantics, performance, security, and operational behavior. Architects should distinguish theoretical openness from tested interchangeability.

7.3 Artificial intelligence and machine learning

AI workloads increase demand for high-quality feature data, vector retrieval, model lineage, and evaluation datasets. An architecture should record training-data versions, feature definitions, model versions, and access decisions. Retrieval-augmented generation systems also require controls for document provenance, prompt injection, sensitive-data leakage, and inaccurate generated answers. AI does not remove the need for governed source data; it magnifies the consequences of poor quality.

7.4 Ethical and societal considerations

Big data systems can reproduce bias, enable intrusive surveillance, or make consequential decisions opaque. Architectural review should ask whether collection is necessary, whether affected people can challenge decisions, whether proxies create discriminatory outcomes, and whether retention is proportionate. Governance must include human accountability, not merely automated policy enforcement.

Best-Practice Checklist

• Begin with business capabilities, workload characteristics, and measurable service levels.
• Separate source-of-record responsibilities from analytical copies.
• Preserve raw data only when purpose, retention, access, and cost are justified.
• Treat data products as supported products with owners and consumers.
• Use contracts, schema compatibility checks, and automated quality tests.
• Prefer replayable and idempotent processing where feasible.
• Design for late data, duplicates, backfills, failure, and recovery from the beginning.
• Apply least privilege, encryption, classification, lineage, and deletion controls.
• Use centralized standards with decentralized expertise when organizational scale demands it.
• Measure actual latency, quality, reliability, and cost after deployment.

Chapter Summary

Big data architecture is the coordinated design of data movement, storage, computation, serving, governance, and organizational responsibility. Warehouses provide governed analytical performance; lakes provide flexible scalable storage; lakehouses combine lake economics with managed analytical tables; meshes organize ownership around domains and data products; and fabrics integrate distributed assets through metadata and automation. These models can be combined, but their assumptions and risks must remain explicit.

Centralized ownership supports consistency and control, while decentralized ownership provides domain knowledge and responsiveness. A hybrid model commonly combines domain accountability with central platform capabilities and federated governance. Batch processing is economical and reproducible; streaming supports low-latency decisions. Lambda offers separate speed and batch paths, whereas Kappa emphasizes replayable stream processing. Neither is universally superior.

Effective design converts vague goals into measurable freshness, availability, recovery, quality, security, and cost requirements. It uses data contracts, explicit ownership, durable and observable pipelines, privacy controls, and tested recovery processes. Contemporary architectures increasingly combine lakehouse storage, event-driven processing, metadata-rich fabrics, domain data products, and AI workloads. The central professional skill is not selecting fashionable technologies but justifying a coherent architecture that remains trustworthy under scale, change, failure, and scrutiny.

Key Terms

Data architecture: The structure and operating model governing data sources, storage, processing, serving, and controls.

Data warehouse: A repository optimized for governed analytical queries over integrated data.

Data lake: Scalable storage for raw or lightly processed data of varied types.

Lakehouse: An architecture combining lake storage flexibility with managed analytical table capabilities.

Data mesh: A domain-oriented paradigm based on ownership, data products, self-service platforms, and federated governance.

Data fabric: A metadata- and integration-centered approach for connecting distributed data environments.

Data product: A documented, governed, supported dataset or data service designed for identified consumers.

Data contract: An agreement defining schema, semantics, quality, compatibility, ownership, and change rules.

Batch processing: Processing data in scheduled or accumulated groups.

Stream processing: Continuous processing of records as they arrive.

Lambda architecture: A pattern combining batch and speed processing paths.

Kappa architecture: A pattern using a replayable stream as the primary processing path.

Event time: The time at which an event actually occurred, as opposed to processing time.

Idempotence: The property that repeated execution produces the same intended result as one execution.

Lineage: The record of data origins, transformations, dependencies, and destinations.

Data quality: The degree to which data is accurate, complete, valid, consistent, unique, and timely for its purpose.

Recovery-time objective: The maximum acceptable time to restore a service after disruption.

Recovery-point objective: The maximum acceptable period of data loss measured backward from failure.

Review Questions

Short Answer Questions

1. Distinguish a data lake from a data warehouse.
2. What are the four principal principles of a data mesh?
3. Why is idempotence important in batch and stream pipelines?
4. Define event time and explain why it matters in stream processing.
5. State two advantages and two risks of centralized data ownership.

Long Answer Questions

1. Compare data warehouses, lakes, lakehouses, meshes, and fabrics, explaining when each is appropriate.
2. Explain Lambda and Kappa architectures, including their processing paths, advantages, and limitations.
3. Develop a governance model for a large organization using decentralized domain ownership.
4. Describe a systematic method for converting business requirements into a big data architecture.
5. Discuss the security, privacy, ethical, and operational controls required for a production data platform.

Analytical / Application Questions

1. A sensor platform receives 20,000 records per second, with an average record size of 500 bytes. Calculate the average raw ingress bandwidth using B = r × s, and identify at least three additional factors that must be included in capacity planning.
2. A bank requires fraud alerts within 5 seconds but performs authoritative regulatory reconciliation overnight. Design a processing architecture and justify the ownership, storage, and batch-versus-streaming choices.
3. An organization has five departments using incompatible definitions of “customer.” Evaluate whether a centralized warehouse, data mesh, data fabric, or hybrid approach should be adopted, and propose controls for semantic consistency.

Multiple Choice Questions with Answers

Question 1

Which architectural model primarily emphasizes domain ownership, data as a product, self-service platforms, and federated governance?

A. Data mesh

B. Data warehouse

C. Lambda architecture

D. Data archive

Correct Answer: A. Data mesh

Question 2

A system processes 900 GB in 30 minutes. What is its average throughput?

A. 0.5 GB/minute

B. 30 GB/minute

C. 270 GB/minute

D. 900 GB/minute

Correct Answer: B. 30 GB/minute

Question 3

Which characteristic is most strongly associated with a data warehouse?

A. Unmanaged storage of arbitrary files

B. Governed analytical querying over integrated data

C. Exclusive use for real-time event alerts

D. Ownership by independent domain teams only

Correct Answer: B. Governed analytical querying over integrated data

Question 4

What is the principal architectural cost of Lambda architecture?

A. It cannot process historical data

B. It requires no serving layer

C. It commonly duplicates processing logic across batch and speed paths

D. It prevents low-latency results

Correct Answer: C. It commonly duplicates processing logic across batch and speed paths

Question 5

Which feature is most important for handling repeated delivery of the same event safely?

A. Idempotence

B. Partition renaming

C. Schema deletion

D. Manual dashboard refresh

Correct Answer: A. Idempotence

Question 6

If 4,000 records arrive per second and each averages 250 bytes, what is the average raw ingress bandwidth before overhead?

A. 1,000 bytes per second

B. 4,250 bytes per second

C. 16 bytes per second

D. 1,000,000 bytes per second

Correct Answer: D. 1,000,000 bytes per second

Question 7

Which statement best describes Kappa architecture?

A. It requires separate batch and speed implementations

B. It uses a replayable stream as the primary computation model

C. It stores only structured warehouse tables

D. It eliminates all batch activities in an organization

Correct Answer: B. It uses a replayable stream as the primary computation model

Question 8

Which responsibility most appropriately belongs to a business data owner?

A. Choosing the physical server’s CPU model

B. Defining business meaning and acceptable use of the data

C. Writing every platform monitoring alert

D. Managing network cabling

Correct Answer: B. Defining business meaning and acceptable use of the data

Question 9

Which control most directly supports traceability from a report back to source data and transformations?

A. Lineage

B. Data compression

C. Load balancing

D. File naming alone

Correct Answer: A. Lineage

Question 10

A requirement states that 95% of valid events must appear in a dashboard within 30 seconds. What kind of requirement is this?

A. A measurable freshness or latency service-level objective

B. A storage encryption algorithm

C. A data ownership assignment

D. A schema serialization format

Correct Answer: A. A measurable freshness or latency service-level objective