Big Data Architecture and Data Life Cycle

Introduction

Big data systems are not defined only by large volumes. They must also accommodate data variety, velocity, variable quality, distributed ownership, and changing analytical requirements. A successful architecture therefore connects the entire path of data: creation, ingestion, storage, processing, serving, analysis, retention, and disposal.

This chapter presents the major architectural patterns used in contemporary big data engineering. It compares data warehouses, data lakes, lakehouses, and data meshes, and explains lambda, kappa, and medallion architectures. The central design principle is that architecture should be selected according to business objectives, latency requirements, data reliability, governance obligations, and operating capability—not according to the popularity of a particular tool.

Learning Objectives

By the end of this chapter, you should be able to:

• describe the stages of a big data life cycle;
• identify the principal components of an end-to-end data architecture;
• distinguish data warehouses, data lakes, lakehouses, and data meshes;
• compare lambda, kappa, and medallion architectures;
• design a suitable architecture for batch, streaming, and analytical workloads;
• evaluate trade-offs involving latency, cost, quality, governance, and maintainability; and
• apply architectural principles to a realistic organizational case.

1. The Big Data Life Cycle

The data life cycle is the controlled movement of data from its point of origin to its eventual archival or deletion. Although diagrams often show a linear sequence, real systems contain feedback loops. Data quality findings may cause changes to ingestion rules; regulatory requests may require locating data in archives; and new analytical requirements may lead to redesigned models.

The principal stages are:

1. Data generation: Data is produced by applications, transactions, sensors, devices, websites, enterprise systems, documents, and external providers.
2. Ingestion: Data is transferred into the platform, either in batches or as a continuous stream.
3. Storage: Data is retained in operational stores, object storage, file systems, databases, warehouses, or specialized repositories.
4. Preparation and processing: Data is validated, cleaned, transformed, joined, enriched, aggregated, and sometimes anonymized.
5. Serving and consumption: Curated data is made available through tables, APIs, dashboards, search systems, feature stores, reports, or analytical notebooks.
6. Archival and retention: Infrequently used data is moved to cheaper storage according to business and legal requirements.
7. Deletion: Data is securely destroyed when its retention period expires or when a valid deletion obligation applies.

At every stage, metadata, security, lineage, quality controls, and access policies should accompany the data. Treating governance as an afterthought is a common cause of unreliable or noncompliant systems.

2. Reference Architecture

A general big data architecture contains the following layers.

2.1 Data sources

Sources may be structured, semi-structured, or unstructured. Examples include relational transaction systems, enterprise resource planning applications, clickstream events, mobile applications, industrial sensors, social media feeds, logs, images, audio, and partner data. Source assessment should consider ownership, schema stability, event time, update frequency, sensitivity, and expected volume.

2.2 Ingestion

Batch ingestion transfers bounded collections at scheduled intervals. It is appropriate for daily financial extracts, periodic snapshots, and systems that cannot expose events continuously.

Streaming ingestion transfers records or events continuously. It is suitable for fraud detection, monitoring, operational alerts, and real-time personalization. A streaming design must address ordering, duplication, late arrival, replay, back-pressure, and failure recovery.

A durable event log can decouple producers from consumers. Producers publish events once, while several independent consumers process those events for storage, monitoring, or alerting. This improves flexibility but introduces requirements for schema compatibility and operational monitoring.

2.3 Storage

Storage should be selected by access pattern rather than by file size alone. Object storage is economical for large immutable or append-oriented datasets. Relational databases are suitable for transactional integrity and highly structured access. Columnar analytical formats are efficient when queries read a subset of columns across many rows. Key-value, document, graph, and time-series databases serve specialized patterns.

Important storage concerns include partitioning, compression, replication, durability, encryption, lifecycle policies, and recovery. Partitioning by date can accelerate time-bounded queries, but excessively small partitions create metadata overhead and slow query planning.

2.4 Processing

Processing engines execute transformations and computations. Batch processing handles bounded data, while stream processing handles unbounded data. Common operations include filtering, projection, joins, windowed aggregation, deduplication, enrichment, and machine-learning feature creation.

Processing semantics matter. At-most-once processing may lose records but avoids duplicates. At-least-once processing avoids silent loss but may produce duplicates. Exactly-once effects require coordinated state, transactional writes, or idempotent operations; they should not be claimed merely because a platform uses the phrase “exactly once.”

2.5 Serving and analytics

The serving layer provides data in forms appropriate to users and applications. Examples include dimensional warehouse tables, materialized views, search indexes, APIs, operational databases, dashboards, and machine-learning feature stores. A single platform rarely optimizes every access pattern, so many architectures use purpose-built serving systems while maintaining shared governance and lineage.

2.6 Cross-cutting controls

Security and governance cross all layers. Relevant controls include identity and access management, encryption in transit and at rest, network isolation, secrets management, audit logging, cataloguing, data classification, quality monitoring, lineage, retention enforcement, and incident response.

A useful reliability measure is availability. If a service is available for time A during an observation period of total time T, then:

Both A and T must use the same time unit. If a service is unavailable for 43.2 minutes in a 30-day month, its availability is approximately 99.9%, because total time is 43,200 minutes and available time is 43,156.8 minutes. Availability alone, however, does not measure correctness, freshness, or completeness.

3. Architectural Styles for Data Management

3.1 Data warehouse

A data warehouse is a managed repository optimized for structured, integrated, and often historical analysis. Data is commonly transformed before loading, a pattern traditionally called extract-transform-load (ETL). Dimensional models with facts and dimensions support consistent reporting and efficient aggregation.

Strengths include strong SQL performance, governed definitions, predictable reporting, and mature access controls. Limitations include the cost of storing large raw datasets, difficulty handling highly irregular data, and the need for modeling before many forms of exploration.

Warehouses are appropriate when an organization needs trusted management reporting, financial controls, standardized metrics, and stable analytical workloads.

3.2 Data lake

A data lake stores raw or lightly processed data in economical, scalable storage. It can accommodate structured, semi-structured, and unstructured data, and often follows an extract-load-transform (ELT) approach. Raw data is valuable because future requirements are not always known at ingestion time.

A lake without governance can become a data swamp: data may be difficult to discover, poorly documented, duplicated, stale, or inaccessible. A sound lake therefore requires catalogs, ownership, quality indicators, naming standards, schema management, lifecycle policies, and controlled zones.

3.3 Lakehouse

A lakehouse combines low-cost, flexible lake storage with warehouse-like management and analytical capabilities. It typically adds transactional consistency, schema enforcement or evolution, versioning, table metadata, and reliable batch and streaming access over object storage.

The lakehouse reduces duplication between a raw lake and an analytical warehouse, but it does not eliminate design problems. Query performance, concurrency, governance, workload isolation, and data modeling still require careful engineering.

3.4 Data mesh

A data mesh is an organizational and architectural approach based on four ideas:

• data is treated as a product with identifiable consumers and quality expectations;
• domain teams own the data products they understand best;
• teams use self-service platform capabilities; and
• common governance is federated across domains.

A mesh is not simply a collection of databases or a replacement for a lake. It addresses organizational scale and ownership. It can improve accountability, but it may also produce inconsistent definitions, duplicated platforms, and coordination costs if standards and interfaces are weak.

These approaches are not mutually exclusive. An organization may use a lakehouse platform, implement domain-owned data products, and expose selected products through a warehouse-style serving layer.

4. Processing Architectures

4.1 Lambda architecture

Lambda architecture separates a batch layer, a speed layer, and a serving layer. The batch layer periodically recomputes comprehensive results from a durable master dataset. The speed layer produces low-latency approximations or incremental results. The serving layer combines them for consumers.

Its advantage is that batch recomputation can correct errors and provide complete historical results. Its principal disadvantage is duplicated logic: transformations may need to be implemented in both batch and streaming technologies. Divergent implementations can produce inconsistent results and increase maintenance cost.

4.2 Kappa architecture

Kappa architecture uses a durable ordered event log and treats both historical recomputation and real-time processing as forms of stream processing. When logic changes, the system replays retained events using the new logic.

Kappa can simplify code by avoiding separate batch and speed implementations. It requires durable event retention, replay capacity, deterministic processing, compatible schemas, and effective state management. It is less suitable when source systems cannot provide complete events or when historical data is available only as periodic snapshots.

4.3 Medallion architecture

Medallion architecture organizes data into progressive quality layers, commonly called bronze, silver, and gold.

• Bronze: raw or minimally altered data, retained for traceability and replay.
• Silver: validated, standardized, deduplicated, and conformed data.
• Gold: business-ready aggregates, data products, semantic models, and serving tables.

The layers describe quality and purpose, not necessarily three physical systems. They should not be interpreted as a guarantee that gold data is correct; quality must be measured and monitored. Quarantine areas are useful for records that fail validation without discarding them.

5. Designing an End-to-End Architecture

Architecture begins with requirements. A practical design sequence is as follows.

Step 1: Identify decisions and users

Start with business decisions rather than technologies. Determine who will use the data, what decisions they make, how often the data changes, and what consequences follow from an incorrect or late result.

Step 2: Characterize workloads

Record volume, velocity, variety, expected concurrency, query patterns, retention, recovery objectives, and latency objectives. Latency should be expressed precisely: for example, “95% of fraud events produce an alert within 5 seconds,” rather than “real time.”

Step 3: Choose ingestion and delivery semantics

Decide whether batch, streaming, or both are required. Define event identifiers, timestamps, replay behavior, deduplication rules, and handling of late data. Idempotent consumers are especially valuable: applying the same event more than once should not change the final result beyond its first application.

Step 4: Establish raw retention and quality zones

Preserve source evidence when legally and economically appropriate. Record ingestion time, source, schema version, and checksum or event identifier. Validate types, required fields, ranges, referential relationships, and business rules before publishing curated data.

Step 5: Select storage and processing

Choose systems based on access patterns, not fashion. Consider total cost, team expertise, portability, operational maturity, security integration, and recovery procedures. Separate compute from storage where elastic scaling is valuable, but account for network and metadata costs.

Step 6: Design serving models

A dashboard may need pre-aggregated tables; an application may need an API or key-value store; a data scientist may need governed files or feature tables. Avoid forcing every consumer to query raw data.

Step 7: Add governance and observability

A production architecture should expose pipeline failures, data freshness, volume changes, schema changes, null rates, duplicate rates, and business-rule violations. Data contracts can specify schemas, ownership, quality expectations, and compatibility rules between producers and consumers.

Step 8: Plan archival and deletion

Retention schedules should distinguish legal obligations from convenience. Archive data using economical storage with tested retrieval procedures. Deletion must include derived copies, indexes, caches, backups where feasible, and downstream data products, subject to applicable law and legitimate preservation requirements.

6. Worked Design Example

Suppose a university wants to analyze student engagement. Learning-management-system events, library transactions, attendance records, and survey responses must support both near-real-time intervention and semester-level reporting.

A suitable design could use streaming ingestion for learning events and batch ingestion for nightly library and attendance extracts. Events enter a durable log and are written to a bronze store with source metadata. Silver processing standardizes student identifiers, validates timestamps, removes duplicates, and applies access restrictions. Gold tables provide weekly engagement measures for institutional reporting, while a low-latency serving store supports alerts when a student’s activity falls sharply.

The architecture should not expose personally identifiable information to every analyst. A governed identity service can maintain the mapping between a protected student identifier and an institutional identity. Analysts may receive pseudonymized records, while authorized advisers access identifiable information for legitimate educational purposes.

This design combines streaming and batch processing, but it does not require a full lambda architecture. A lakehouse can provide storage and analytical tables, while medallion layers describe data refinement. Domain ownership may follow a data-mesh principle: the library, learning, and student-support domains publish documented data products under common institutional governance.

7. Illustrative Case Study: Retail Fraud Detection

Background

An online retailer processes orders, payment events, device signals, and account changes. Fraud patterns can emerge within seconds, but finance teams also need complete daily reconciliation.

Problem

The retailer’s previous nightly warehouse load detected fraud too late. A proposed streaming solution reduced latency but produced inconsistent results because the streaming and batch rules differed.

Analysis

Requirements were defined as follows: high-risk transactions should be scored within 3 seconds; all events must be retained for 13 months; duplicate events must not inflate transaction counts; and analysts must be able to replay history after a rule change.

Approach

The retailer adopted a durable event-log and lakehouse design. Events received unique transaction and event identifiers. A streaming processor generated provisional risk scores and wrote them to a serving database. Raw events were retained in the bronze layer. Silver processing standardized events and applied deterministic deduplication. Gold tables supported finance reconciliation and model analysis.

The organization chose a hybrid operating model rather than blindly implementing a named architecture. Streaming supported immediate decisions, while scheduled recomputation validated historical results. Shared transformation definitions and automated comparison tests reduced divergence between paths. Sensitive payment data was tokenized, access was role-based, and audit logs recorded use of high-risk attributes.

Outcome

The retailer obtained low-latency intervention and reliable historical reporting. It also accepted that provisional streaming results could be revised when late events arrived. Dashboards therefore displayed both event freshness and result status.

Learning points

The case demonstrates that latency, completeness, replay, and governance must be designed together. It also shows why architecture labels are less important than explicit requirements and testable guarantees.

8. Governance, Ethics, and Regulation

Big data architectures can amplify unfairness and privacy risks. Data minimization asks whether each collected attribute is necessary. Purpose limitation asks whether data is used consistently with the reason for collection. Fairness assessment is required when models or rules affect access to credit, education, employment, healthcare, or public services.

Important practices include classification of sensitive data, least-privilege access, separation of duties, consent or other lawful basis where applicable, explainable policies, retention limits, breach response, and documented lineage. Organizations should distinguish technical possibility from ethical legitimacy. A system may be able to infer a sensitive characteristic without having a defensible reason to do so.

Artificial intelligence increases the importance of architecture. Training data must be versioned, features must be traceable, and models require monitoring for drift, bias, degradation, and unauthorized use. Generative AI applications additionally require controls for confidential prompts, provenance, hallucinations, and human review.

9. Challenges and Best Practices

Common challenges include uncontrolled schema changes, duplicate data, late events, expensive small-file patterns, unclear ownership, insufficient metadata, vendor lock-in, and hidden dependencies between pipelines. Technical debt grows when teams optimize only for an initial demonstration.

Best practices include:

• define data products, owners, consumers, and service expectations;
• use schema versioning and compatibility checks;
• retain raw evidence where appropriate and document transformations;
• make writes idempotent and design for replay;
• separate sensitive identity data from analytical attributes;
• automate data-quality and infrastructure tests;
• monitor freshness, completeness, accuracy indicators, and cost;
• use partitioning and compaction based on measured workload behavior;
• test disaster recovery and archival retrieval; and
• prefer the simplest architecture that satisfies verified requirements.

Chapter Summary

Big data architecture connects sources, ingestion, storage, processing, serving, analytics, archival, and deletion. Batch and streaming ingestion solve different timing problems, while storage and processing systems should be selected according to workload and access pattern. Warehouses emphasize governed structured analytics; lakes emphasize flexible economical retention; lakehouses combine lake storage with warehouse-like controls; and data meshes organize ownership around domain data products and federated governance.

Lambda architecture combines batch and speed paths, Kappa architecture uses replayable event processing, and medallion architecture organizes data by progressive refinement. These patterns are complementary rather than universally competing. Effective designs begin with users, decisions, latency, quality, security, retention, and recovery requirements. Governance, privacy, observability, and lifecycle management are architectural concerns, not optional additions.

Key Terms

Data life cycle: The stages through which data is generated, processed, used, retained, archived, and deleted.

Batch processing: Processing a bounded collection of data at a scheduled or triggered time.

Stream processing: Continuous processing of records or events as they arrive.

Data warehouse: A structured repository optimized for governed analytical queries.

Data lake: Scalable storage for raw or lightly processed heterogeneous data.

Lakehouse: An architecture combining lake storage flexibility with warehouse-style management and analytics.

Data mesh: A domain-oriented approach based on data products, domain ownership, self-service platforms, and federated governance.

Data product: A maintained, documented data asset designed for identifiable consumers.

Lambda architecture: A pattern combining batch, speed, and serving layers.

Kappa architecture: A replay-oriented architecture based primarily on stream processing.

Medallion architecture: A layered organization commonly using bronze, silver, and gold data zones.

Data contract: An explicit agreement about schema, semantics, quality, ownership, and compatibility.

Idempotence: The property that repeated application of an operation produces the same intended result as one application.

Data lineage: A record of data origins, transformations, movements, and destinations.

Schema evolution: Controlled change to the structure or meaning of data over time.

Event time: The time at which an event actually occurred, as distinct from ingestion or processing time.

Data governance: Policies, roles, controls, and practices for managing data responsibly.

Data swamp: A poorly governed data lake that is difficult to discover, trust, or use.

Data archival: Movement of less frequently used data to lower-cost retention storage.

Review Questions

Short Answer Questions

1. What are the principal stages of the big data life cycle?
2. How does streaming ingestion differ from batch ingestion?
3. State one major strength and one limitation of a data warehouse.
4. What is the purpose of the silver layer in a medallion architecture?
5. Why are metadata and lineage important in a big data platform?

Long Answer Questions

1. Compare data lakes, warehouses, lakehouses, and data meshes, including situations in which each is appropriate.
2. Explain lambda and Kappa architectures and discuss the maintenance trade-offs between them.
3. Describe how security and governance should be implemented across an end-to-end data architecture.
4. Explain the role of replay, deduplication, event time, and late-arriving data in streaming systems.
5. Design an architecture for an organization requiring both daily regulatory reporting and near-real-time operational alerts.

Analytical / Application Questions

1. A company has duplicated transformation logic in batch and streaming pipelines, producing different sales totals. Diagnose the likely causes and propose a testing and architectural response.
2. An organization wants to retain raw customer data indefinitely because future uses are unknown. Evaluate this proposal using cost, privacy, governance, and regulatory considerations.
3. A university wants domain teams to publish data products but has no common definitions for student, course, or enrollment. Analyze the risks and propose a federated governance model.

Multiple Choice Questions with Answers

Question 1

Which architecture combines raw, validated, and business-ready data zones commonly called bronze, silver, and gold?

A. Medallion architecture

B. Kappa architecture

C. Star schema

D. Peer-to-peer architecture

Correct Answer: A. Medallion architecture

Question 2

What is the principal purpose of a durable event log in a streaming architecture?

A. To replace all analytical databases

B. To decouple producers and consumers and support replay

C. To guarantee that no event can ever be duplicated

D. To eliminate the need for schema management

Correct Answer: B. To decouple producers and consumers and support replay

Question 3

Which characteristic most directly distinguishes a data warehouse from a data lake?

A. A warehouse is generally optimized for governed structured analytics

B. A warehouse can store only numerical data

C. A lake cannot contain historical data

D. A lake always processes data faster than a warehouse

Correct Answer: A. A warehouse is generally optimized for governed structured analytics

Question 4

In Kappa architecture, historical recomputation is primarily performed by:

A. Deleting the event history

B. Replaying retained events through stream processing

C. Loading only the latest database snapshot

D. Using a separate speed layer with no source log

Correct Answer: B. Replaying retained events through stream processing

Question 5

Which property means that processing the same event repeatedly does not incorrectly change the final result?

A. Partition tolerance

B. Idempotence

C. Normalization

D. Serialization

Correct Answer: B. Idempotence

Question 6

A service is unavailable for 43.2 minutes during a 30-day month. Its approximate availability is:

A. 90.0%

B. 99.0%

C. 99.9%

D. 99.99%

Correct Answer: C. 99.9%

Question 7

Which data-mesh principle assigns primary responsibility for a data product to the domain that understands it best?

A. Centralized ownership by one database team

B. Domain-oriented ownership

C. Anonymous ownership

D. Ownership only by external vendors

Correct Answer: B. Domain-oriented ownership

Question 8

Why can a data lake become a data swamp?

A. Because object storage cannot store large files

B. Because insufficient governance makes data difficult to discover and trust

C. Because all lake data must be relational

D. Because lakes cannot support metadata

Correct Answer: B. Because insufficient governance makes data difficult to discover and trust

Question 9

Which requirement is most specifically associated with handling late-arriving streaming records?

A. Defining event-time and correction or update behavior

B. Removing all timestamps from events

C. Prohibiting replay of historical data

D. Using only daily batch ingestion

Correct Answer: A. Defining event-time and correction or update behavior

Question 10

Which is an appropriate cross-cutting governance control?

A. Disabling audit logs to improve query speed

B. Granting every analyst unrestricted access

C. Maintaining lineage, classification, and least-privilege access

D. Deleting source data immediately after ingestion

Correct Answer: C. Maintaining lineage, classification, and least-privilege access