Data Engineering, Big Data and Cloud Foundations

Introduction

Data science rarely succeeds through modelling alone. A useful model depends on data that is discoverable, correctly interpreted, timely, secure and available at the required scale. Data engineering provides the systems, processes and controls that move data from operational sources to analytical and machine-learning environments. Big-data technologies address situations in which data volume, velocity, variety or complexity exceeds the practical capabilities of traditional systems. Cloud platforms provide elastic computing, managed storage and network-accessible services that allow organisations to build such systems without owning all physical infrastructure.

This chapter examines the relationship between data engineering and data science. It explains ETL and ELT pipelines, warehouses, lakes and lakehouses, distributed processing, batch and streaming architectures, cloud foundations, scalability, governance and team collaboration. The emphasis is not on memorising product names but on understanding architectural decisions and their consequences.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish data engineering from data science and data analytics.
2. Design and evaluate ETL and ELT pipelines.
3. Compare data warehouses, data lakes and lakehouses.
4. Explain distributed processing, partitioning and fault tolerance.
5. Distinguish batch, micro-batch and real-time streaming systems.
6. Describe core cloud service models and scalability principles.
7. Select appropriate quality, reliability, security and governance controls.
8. Explain how engineering and data-science teams collaborate throughout the model lifecycle.

1. Data Engineering as an Organisational Capability

Data engineering is the discipline of designing, implementing and operating systems that collect, store, transform, serve and protect data. A data engineer is concerned with data availability, correctness, lineage, cost, performance and operational reliability. A data scientist generally uses prepared data to investigate questions, construct models and communicate findings. These roles overlap, but they have different primary objectives.

A typical data lifecycle is:

source → ingestion → storage → transformation → quality validation → serving → analysis or modelling → monitoring and governance

Sources may include relational databases, application logs, sensors, files, third-party APIs and event streams. Ingestion can be scheduled or event-driven. Storage may be operational, analytical or archival. Serving means making reliable data available through tables, APIs, feature stores, dashboards or model-serving systems.

A useful engineering principle is that data is a product, not merely a by-product. A data product has intended users, documented meaning, measurable quality, an owner and a support process. For example, “daily active customers” should have a precise definition, refresh time, inclusion rules and a known responsible team.

Important quality dimensions

A pipeline that produces data quickly but inaccurately is not successful. Quality requirements should therefore be defined before implementation, ideally as service-level objectives (SLOs), such as “95% of daily sales data is available by 06:00 and has fewer than 0.1% invalid transaction records.”

2. ETL and ELT Pipelines

ETL

ETL means extract, transform, load. Data is extracted from source systems, transformed in an intermediate processing environment and loaded into a target warehouse or analytical store. Traditional ETL was common when target systems had limited processing capacity and transformations had to be completed before loading.

A transformation may standardise dates, remove duplicates, join reference data, aggregate transactions or encode business rules. ETL can protect a warehouse from unsuitable raw data, but it may discard original detail unless raw copies are retained.

ELT

ELT means extract, load, transform. Data is loaded, often in relatively raw form, into a scalable analytical platform; transformations are then executed there. Cloud warehouses and lakehouse engines make ELT attractive because storage is inexpensive and compute can be provisioned when needed. ELT supports reprocessing when definitions change, provided raw data and metadata have been retained.

The distinction is architectural rather than absolute. Modern systems often use a hybrid pattern: validate and lightly transform data before loading, then perform business transformations in the warehouse or lakehouse.

Pipeline stages

1. Extraction or capture: Obtain records through queries, files, APIs, database logs or events.
2. Ingestion: Transfer data reliably, often using queues or object storage.
3. Schema handling: Record expected fields, data types, versions and compatibility rules.
4. Transformation: Clean, join, enrich, aggregate or reshape data.
5. Validation: Check constraints, counts, ranges, freshness and referential relationships.
6. Publication: Expose approved datasets to analysts, applications or models.
7. Monitoring: Track failures, latency, cost, quality and lineage.

A robust pipeline should be idempotent: rerunning the same logical input should not create unintended duplicates or inconsistent results. This may be achieved through deterministic transformations, unique event identifiers, merge operations and checkpoints.

Batch, incremental and change-data-capture processing

Batch processing handles a bounded collection of data at scheduled intervals. It is suitable for payroll, daily reporting and many training-data workflows. Incremental processing handles only new or changed records, reducing computation compared with rebuilding an entire dataset.

Change data capture (CDC) records inserts, updates and deletes from an operational database, commonly by reading transaction logs. CDC can reduce source-system load and support near-real-time replication. However, engineers must preserve event ordering where required, handle late-arriving changes and define how deletions are represented.

A simple throughput relationship is:

where T is processing time, D is the amount of data and R is sustained processing rate. The relationship assumes a stable rate and ignores startup, coordination, network and skew overhead. For example, processing 720 GB at 120 GB per hour requires approximately 6 hours under these idealised assumptions. In practice, partitioning and parallelism may change the effective rate, while bottlenecks may make the actual time longer.

3. Analytical Storage: Warehouses, Lakes and Lakehouses

Data warehouses

A data warehouse is a curated analytical repository designed for structured data, consistent definitions and efficient queries. Warehouses commonly use column-oriented storage, compression, indexing or clustering, and SQL interfaces. They are strong for governed reporting and dimensional modelling.

A dimensional warehouse often contains fact tables, which record measurable events, and dimension tables, which describe entities such as customers, products or dates. A sales fact might contain quantity, revenue, product key, customer key and date key. A star schema places a fact table at the centre and dimensions around it. This can make analytical queries understandable and performant.

Warehouses usually prioritise conformity and controlled access. Their limitations may include ingestion cost, reduced flexibility for unstructured data and friction when storing raw data for future unknown uses.

Data lakes

A data lake stores large quantities of raw, semi-structured and structured data, often in object storage. It can retain logs, images, documents, JSON events and relational extracts. A lake supports exploration and machine learning, but raw flexibility can become a liability if metadata, ownership and quality controls are weak. The term data swamp describes a lake in which users cannot reliably discover or interpret its contents.

Lakes commonly use zones such as:

• Raw or bronze: Source-faithful data, minimally altered.
• Standardised or silver: Cleaned, typed and conformed data.
• Curated or gold: Business-ready datasets, metrics and features.

These labels are conventions rather than universal technical standards.

Lakehouses

A lakehouse combines low-cost, flexible object storage with warehouse-like management, transactional consistency, schema enforcement, table metadata and analytical performance. It aims to avoid maintaining separate copies for every analytical use. In practice, a lakehouse is a design pattern supported by table formats and processing engines; specific capabilities vary by platform.

The choice among these architectures depends on data types, latency, governance, workload, team skills and cost. It is misleading to ask which is universally best. A regulated financial report may require highly curated warehouse tables, while exploratory image analysis may begin in a governed lake.

Figure: Conceptual comparison of warehouse, lake and lakehouse characteristics.

4. Distributed Processing and Big Data

Big data is not defined only by size. The commonly cited dimensions are volume, velocity and variety; veracity, value and variability are often added. A dataset may be “big” because it arrives rapidly, has complex formats, requires extensive computation or must be retained for many users.

Distributed processing divides data and computation across multiple machines. A typical framework separates a logical job into stages. Data is partitioned, tasks operate in parallel, intermediate results may be shuffled across the network, and outputs are written to distributed storage.

Partitioning and parallelism

Partitioning assigns records to subsets, often by key or file range. Good partitions permit parallel work and reduce unnecessary data movement. Poor partitions create data skew, in which one task receives much more work than others. The fastest workers then wait for the slowest partition.

If a task processes an amount of data D_i at rate R_i, an idealised task time is t_i = D_i ÷ R_i. For a parallel stage, completion is governed approximately by the slowest task, so T_stage ≈ max(t_i) plus coordination and transfer overhead. This explains why adding machines does not always produce proportional speed improvement.

Fault tolerance and consistency

Distributed systems expect machine or network failures. Fault tolerance may use replicated data, task retries, checkpoints and recomputation. Exactly-once processing is not automatically guaranteed merely because a platform advertises it; the complete system, including source, transport, sink and recovery logic, must support the required semantics.

A useful distinction is:

• At-most-once: A record may be lost, but is not processed more than once.
• At-least-once: A record is not intentionally lost, but duplicates may occur.
• Exactly-once effect: The final observable result behaves as if each record affected the result once, usually through transactions, deduplication or idempotent sinks.

The appropriate choice depends on business consequences. Duplicate advertising impressions may be tolerable after deduplication, whereas duplicate financial transfers are unacceptable.

5. Streaming Data Systems

Streaming systems process continuously arriving events rather than waiting for a complete batch. Examples include payment authorisations, sensor measurements, website clicks and application logs. A stream architecture commonly includes producers, a durable event broker, consumers, state management and an output store.

Important concepts include:

• Event time: When the event occurred.
• Processing time: When the system handled the event.
• Ingestion time: When the platform received the event.
• Window: A bounded interval used for aggregation.
• Watermark: An operational estimate of how complete event-time data is up to a point.
• Late event: An event that arrives after the system has advanced past its event-time position.

A tumbling window has fixed, non-overlapping intervals. A sliding window may move in smaller increments and overlap. A session window groups activity separated by less than a specified inactivity gap. Engineers must define lateness rules, correction behaviour and retention of state.

Streaming is not automatically preferable to batch. Real-time processing increases operational complexity, monitoring requirements and often cost. Batch is suitable when decisions do not require immediate data. A hybrid architecture may use streaming for alerts and batch for authoritative daily reconciliation.

6. Cloud Foundations and Scalability

Cloud computing provides on-demand access to shared computing resources, commonly with measured usage and elastic provisioning. Three broad service models are:

Cloud data architectures usually combine object storage, managed databases or warehouses, compute engines, networking, identity services, monitoring and orchestration. Shared responsibility means the provider secures the cloud infrastructure, while the customer remains responsible for choices such as access policies, data classification, credentials and correct configuration. The exact boundary differs by service.

Vertical and horizontal scaling

Vertical scaling increases the resources of one machine, such as CPU or memory. It is simple but limited by machine capacity and may require downtime. Horizontal scaling adds machines or workers. It can provide greater capacity but introduces coordination, partitioning and distributed-failure concerns.

Amdahl’s law expresses a limit on speedup when only part of a workload can be parallelised:

where S(N) is speedup using N processors, and p is the fraction of execution time that is parallelisable. The assumptions are a fixed workload, ideal processor availability and no additional parallel overhead. If p = 0.8 and N = 4, then S(4) = 1 ÷ (0.2 + 0.8 ÷ 4) = 1 ÷ 0.4 = 2.5. Thus four processors do not produce fourfold speedup because 20% remains serial. Real systems usually perform worse because of communication, scheduling and storage overhead.

Elasticity and cost

Elasticity means resources can expand or contract in response to demand. It is valuable for irregular workloads such as model training or month-end reporting. However, elastic systems require budgets, quotas, automated shutdown, workload tagging and cost monitoring. The cheapest architecture is not necessarily the one with the lowest hourly price; engineers should consider storage, data transfer, idle resources, operational labour and failure costs.

7. Architecture, Governance and Security

A sound architecture separates concerns while preserving traceability. Common architectural principles include:

• Loose coupling: Components communicate through stable interfaces or events.
• Single source of truth: Important metrics have authoritative definitions.
• Metadata-first design: Users can discover ownership, meaning, lineage and quality.
• Least privilege: Users and services receive only the access they require.
• Defence in depth: Security uses multiple controls rather than one barrier.
• Reproducibility: Data versions, code, configuration and dependencies are recorded.
• Observability: Logs, metrics and traces reveal system behaviour.

Governance includes policies for retention, classification, access, quality, lineage, consent, acceptable use and deletion. Personally identifiable information (PII), health information and financial data may be subject to laws and sector regulations. Compliance is not achieved by putting data in the cloud or by anonymising a column without assessing re-identification risk.

Useful controls include encryption in transit and at rest, role-based access, secret management, network isolation, audit logs, tokenisation, masking and retention schedules. Data minimisation is ethically important: collecting data merely because it might become useful increases exposure and governance burden.

Models create additional obligations. Training data may encode historical discrimination; a pipeline may silently change the population represented; and a feature may leak future information into training. Data engineers and data scientists should document provenance, permitted use, sampling decisions and known limitations.

8. Collaboration Between Data Science and Engineering

Collaboration should begin before modelling. Data scientists contribute business questions, target definitions, exploratory insight and modelling requirements. Data engineers contribute source-system knowledge, reliable pipelines, scalable computation, security controls and operational support. Product owners, domain specialists, security professionals and governance officers may also be essential.

A practical workflow is:

1. Define the decision, population, target and success criteria.
2. Inventory sources and assess legal and ethical permissions.
3. Create a data contract specifying fields, types, meanings, freshness and change rules.
4. Build a reproducible ingestion and transformation pipeline.
5. Validate quality with automated tests and representative samples.
6. Develop features and models using versioned data.
7. Deploy through controlled interfaces and monitor both system and model behaviour.
8. Feed observations back into pipeline and model improvement.

A data contract is an explicit agreement between a producing system and its consumers. It can specify schema, nullability, units, allowed values, event identifiers, delivery expectations and compatibility policy. Contracts reduce surprise failures when operational teams change applications.

In production, monitoring should include pipeline failures, data freshness, volume anomalies, schema changes, distribution drift, prediction performance and business outcomes. A model can remain technically available while becoming unsuitable because customer behaviour, policy or upstream data has changed.

9. Illustrative Case: Real-Time Fraud Detection for a Digital Bank

Background

A digital bank processes card transactions through mobile applications, merchants and payment networks. It wants to identify suspicious transactions within seconds while retaining a reliable daily record for regulatory reporting.

Problem

The bank’s existing nightly ETL process detects fraud too late. Transaction data comes from several systems with different customer identifiers. Some events arrive late, and duplicate messages occur when network retries happen. Fraud analysts also need explanations and an audit trail.

Analysis

The decision requires low latency, but regulatory reporting requires completeness and reconciliation. A purely real-time system could produce rapid decisions but might omit late or corrected events. A purely batch system is too slow. The architecture therefore needs both streaming and batch paths, with a shared event identity and authoritative reconciliation process.

Approach and solution

Transaction events are published to a durable broker with a unique transaction identifier, event timestamp and schema version. A streaming consumer validates required fields, enriches events with customer and merchant features, and applies a fraud model. Idempotent storage ensures that retries do not create multiple financial effects. A short event-time window handles related transactions, while a rule identifies unusually rapid spending.

Raw events are also written to governed object storage. A batch process reconciles the raw event history with settlement records, handles late arrivals and produces curated warehouse tables. Data quality tests compare transaction counts and monetary totals across stages. Access to identity fields is restricted, and analysts receive masked data where possible.

The data-science team versions training datasets, features and model artefacts. The engineering team deploys the scoring service, manages scaling and monitors latency. Both teams define escalation procedures for model drift and false-positive rates. Fraud analysts review sampled decisions, and their feedback is recorded with appropriate controls.

Outcome

The bank obtains rapid provisional decisions and a complete reconciled record for reporting. The design accepts that streaming output may be corrected by later information. This distinction between immediate operational action and authoritative historical truth is central to the architecture.

Learning points

1. Latency and completeness are different requirements.
2. Event identifiers and idempotency are essential when retries occur.
3. Batch reconciliation can complement, rather than replace, streaming.
4. Governance must be designed with the data path, not added afterwards.
5. Model performance and pipeline performance must be monitored together.

10. Challenges, Trade-offs and Contemporary Directions

Data platforms can fail through complexity as well as through technical defects. Excessive copies increase cost and create inconsistent definitions. Overly centralised systems can create bottlenecks, while uncontrolled decentralisation creates duplicated tools and fragmented governance. A platform should be evaluated against actual workload and organisational capability.

Common failure modes include loading data without ownership, treating dashboards as authoritative without lineage, ignoring deletion requests, partitioning on high-cardinality values without analysis, and selecting streaming technology when daily processing is sufficient.

Contemporary developments include serverless data processing, open table formats, data mesh practices, data contracts, vector databases and retrieval-augmented AI systems. These developments do not remove foundational responsibilities. AI applications still require trustworthy ingestion, access control, evaluation data, monitoring and protection against sensitive-data leakage. A vector index, for example, is not a substitute for source-data governance or accurate metadata.

Best practice is therefore contextual: begin with business decisions and service requirements; choose the simplest architecture that meets them; retain provenance; automate quality checks; secure data by design; and test operational failure modes before production.

Chapter Summary

Data engineering creates the reliable infrastructure through which data becomes usable for analytics and machine learning. ETL transforms data before loading, whereas ELT loads data before transforming it; hybrid approaches are common. Warehouses provide curated analytical structures, lakes provide flexible storage for diverse data, and lakehouses seek to combine flexibility with transactional and governance capabilities.

Distributed processing uses partitioning and parallel execution, but speedup is limited by serial work, data skew and coordination overhead. Streaming systems support rapid decisions and require explicit treatment of event time, late data, windows and delivery semantics. Cloud platforms offer managed services and elasticity, but customers remain responsible for configuration, access and data governance.

Effective architectures combine quality controls, metadata, lineage, security, observability and reproducibility. Data scientists and engineers must collaborate through data contracts, versioned workflows and shared monitoring. In professional practice, the best architecture balances latency, completeness, cost, reliability, ethical responsibility and organisational capability.

Key Terms

Data engineering: Design and operation of systems that collect, transform, store and serve data.

ETL: Extract, transform and load processing pattern.

ELT: Extract, load and transform processing pattern.

Idempotency: Property whereby repeating an operation produces the same intended result without unintended duplication.

Change data capture: Technique for recording inserts, updates and deletes from a source system.

Data warehouse: Governed analytical repository, usually optimised for structured queries.

Data lake: Flexible repository for raw and varied data, often based on object storage.

Lakehouse: Architecture combining lake-style storage with warehouse-like management and analytics.

Fact table: Dimensional-model table containing measurable events.

Dimension table: Table describing entities or contextual attributes in a dimensional model.

Partitioning: Dividing data or computation into subsets for storage or parallel processing.

Data skew: Uneven distribution of work across distributed partitions.

Fault tolerance: Ability to continue or recover when components fail.

Event time: Time at which an event occurred.

Watermark: Estimate of how complete event-time processing is up to a point.

Elasticity: Ability to expand or contract resources in response to demand.

Data contract: Agreement defining data schema, meaning, quality and change expectations.

Data lineage: Record of data origins, transformations and destinations.

Data drift: Change in the statistical properties or meaning of data over time.

Review Questions

Short Answer Questions

1. What are the main differences between ETL and ELT?
2. Why is idempotency important in data pipelines?
3. Distinguish a data warehouse from a data lake.
4. What is data skew in a distributed processing system?
5. Define event time and explain why it matters in streaming systems.

Long Answer Questions

1. Explain the lifecycle of a governed analytical dataset from source capture to publication.
2. Compare warehouses, lakes and lakehouses in terms of storage flexibility, governance and workload suitability.
3. Explain how partitioning, parallelism and fault tolerance interact in distributed processing.
4. Discuss the shared-responsibility model in cloud data platforms and its security implications.
5. Describe a professional collaboration process between data scientists and data engineers for deploying a machine-learning system.

Analytical / Application Questions

1. An organisation receives 10 TB of logs daily but only needs a refreshed compliance report each morning. Recommend a processing architecture and justify whether streaming is necessary.
2. A distributed job has one partition containing 40% of all records while the remaining partitions are similar and small. Explain the likely performance effect and propose two remedies.
3. Design a combined batch-and-streaming architecture for detecting suspicious online payments. Specify how you would address duplicates, late events, data quality, security and reconciliation.

Multiple Choice Questions with Answers

Question 1

Which statement best describes ELT?

A. Data is extracted, transformed in an intermediate server, and then loaded.

B. Data is extracted, loaded into a target platform, and transformed there.

C. Data is only transformed after a machine-learning model is deployed.

D. Data is encrypted, logged, and transferred without transformation.

Correct Answer: B. Data is extracted, loaded into a target platform, and transformed there.

Question 2

Which property ensures that rerunning a pipeline does not create unintended duplicate results?

A. Elasticity

B. Partitioning

C. Idempotency

D. Virtualisation

Correct Answer: C. Idempotency

Question 3

Which repository is generally most associated with curated structured analytical data and governed SQL reporting?

A. Data warehouse

B. Message broker

C. Raw data lake zone

D. Operational cache

Correct Answer: A. Data warehouse

Question 4

In a distributed computation, what is a likely consequence of severe data skew?

A. All workers finish at exactly the same time

B. One or more workers become bottlenecks

C. The data automatically becomes encrypted

D. The workload requires no network communication

Correct Answer: B. One or more workers become bottlenecks

Question 5

A stream processor uses the time at which a transaction actually occurred, rather than the time it was received. Which concept is being used?

A. Processing time

B. Ingestion time

C. Event time

D. Wall-clock partitioning

Correct Answer: C. Event time

Question 6

Under the basic IaaS model, which responsibility normally remains primarily with the customer?

A. Manufacture of physical servers

B. Operation of the provider’s data-centre power system

C. Configuration of the customer’s operating system and applications

D. Maintenance of the cloud provider’s hypervisor

Correct Answer: C. Configuration of the customer’s operating system and applications

Question 7

Using S(N) = 1 ÷ ((1 − p) + p ÷ N), what is the idealised speedup when p = 0.8 and N = 4?

A. 1.25

B. 2.00

C. 2.50

D. 4.00

Correct Answer: C. 2.50

Question 8

Which practice most directly supports discoverability and accountability for an analytical dataset?

A. Removing all metadata to reduce storage

B. Recording ownership, meaning, lineage and quality information

C. Allowing every user unrestricted access

D. Keeping only the latest transformed output

Correct Answer: B. Recording ownership, meaning, lineage and quality information

Question 9

Why might an organisation combine streaming with batch processing for financial transactions?

A. Streaming guarantees that no data needs to be stored

B. Batch processing always has lower latency than streaming

C. Streaming supports rapid decisions while batch can reconcile late or corrected records

D. Batch processing eliminates the need for data governance

Correct Answer: C. Streaming supports rapid decisions while batch can reconcile late or corrected records

Question 10

What is the primary purpose of a data contract?

A. To replace all security controls with documentation

B. To define schema, meaning, quality and change expectations between producers and consumers

C. To guarantee that every model is accurate

D. To convert unstructured data into images

Correct Answer: B. To define schema, meaning, quality and change expectations between producers and consumers