12. Data Integration, ETL/ELT and Data Quality

Introduction and Context

Most organisations do not operate from a single, uniform database. Customer information may be maintained in a CRM system, orders in an enterprise resource planning system, payments by a banking service, website activity in event logs, and operational events through message brokers. Data science and managerial reporting become reliable only when these heterogeneous sources are connected, interpreted consistently, and monitored over time. Data integration is therefore not merely a technical exercise of moving records. It is the disciplined process of making data from different origins available as a coherent, timely, traceable, and usable resource.

The traditional approach to integration is extract, transform, and load (ETL). Data is extracted from operational systems, transformed into an appropriate structure and meaning, and loaded into a warehouse or other analytical store. Modern cloud platforms have encouraged extract, load, and transform (ELT), in which relatively raw data is loaded into a scalable target and transformed there. Both approaches remain useful. The choice depends on data volume, latency requirements, governance, computational architecture, security, and the intended use of the data.

Integration also includes streaming pipelines, application programming interfaces (APIs), messaging systems, and change data capture (CDC). These mechanisms address different temporal needs. A monthly financial report may require a batch process, whereas fraud detection may require events to be processed within seconds. In both cases, data quality is central. A fast pipeline that produces duplicated, incomplete, or semantically inconsistent data can create more damage than a slow pipeline that is transparent and controlled.

This chapter examines the architecture and practice of batch and streaming integration, ETL and ELT, CDC, APIs, messaging, metadata, lineage, master data management, validation, observability, and data-quality measurement. It treats the pipeline as an organisational information system whose outputs require technical design, business ownership, governance, and continuous improvement.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the role of data integration in analytical and operational environments.

2. Distinguish ETL, ELT, batch, micro-batch, and streaming architectures.

3. Design an integration workflow using databases, APIs, CDC, and messaging systems.

4. Select suitable methods for handling schema changes, duplicates, late events, and failures.

5. Explain metadata, data lineage, master data, and data contracts.

6. Define and calculate practical data-quality dimensions and metrics.

7. Apply validation, observability, governance, and remediation practices to pipelines.

8. Evaluate the organisational, ethical, security, and managerial implications of integrated data.

Data Integration as a Systems Problem

Data integration has three related objectives. The first is syntactic integration: bringing data together despite differences in file formats, database schemas, encodings, and protocols. The second is semantic integration: ensuring that apparently similar fields have the same meaning. For example, “customer,” “active account,” and “billable subscriber” may refer to different populations. The third is operational integration: delivering data with an appropriate frequency, reliability, and level of traceability.

A useful integration architecture separates source systems, ingestion, storage, transformation, serving, and governance. Source systems are designed primarily for transactions. The ingestion layer captures files, records, API responses, or events. A landing or raw zone preserves received data, while curated layers apply standardisation and business rules. Serving layers provide datasets, dashboards, feature stores, or operational APIs. Governance spans all layers rather than being added at the end.

Integration is often described as a pipeline, but modern platforms are better understood as data products and data flows. A pipeline has inputs, processing rules, dependencies, outputs, and operational controls. A data product additionally has an owner, documented purpose, quality expectations, access rules, and users. This distinction is important because a technically successful load may still fail its organisational purpose if the data is misunderstood or cannot be trusted.

Important Terminology

Source system is the application or repository in which data is originally created or maintained. A target system is the destination to which integrated data is delivered. An ingestion process acquires data from a source. A pipeline is an ordered set of activities that moves and processes data. Latency is the time between data creation and availability for use. Throughput is the amount of data processed per unit of time. Idempotence means that repeating an operation produces the same intended result rather than additional unintended effects. Schema describes the structure, types, constraints, and relationships of data. Schema drift occurs when a source structure changes over time. Data contract is an explicit agreement between a producer and consumers concerning schema, meaning, quality, and change procedures.

ETL and ELT: Two Patterns for Transformation

In an ETL architecture, extraction obtains data from one or more sources, transformation cleans and reshapes it, and loading writes the result to the target. Transformations may include type conversion, standardisation of dates and codes, joining reference data, filtering invalid records, deduplicating entities, and calculating measures. ETL is valuable when the target requires only approved data, when transformation must occur before data enters a controlled environment, or when source data must be reduced before transfer.

ELT reverses the final two stages. Data is extracted and loaded, often into a cloud data lakehouse or warehouse, and transformations are executed using the target platform’s compute resources. ELT supports preservation of raw data, rapid onboarding of new sources, and reproducible transformation through version-controlled SQL or code. It can, however, increase storage and governance demands. Raw data may contain personal or confidential information, and unrestricted transformation flexibility can result in multiple conflicting definitions of the same metric.

The distinction is not absolute. Organisations commonly use a hybrid model: lightweight transformations during ingestion, raw retention in a protected landing zone, and business transformations in the warehouse. The appropriate choice should be based on measurable requirements rather than on the popularity of a tool or architectural label.

A reliable batch workflow normally includes source assessment, extraction, landing, validation, transformation, loading, reconciliation, and publication. Extraction should record the source, extraction time, batch identifier, and record counts. The landing stage should preserve the received representation where policy permits. Transformation should be deterministic and documented. Loading should use transactions, staging tables, merge operations, or partition replacement so that partially completed jobs do not appear as complete data. Reconciliation compares source and target counts, totals, keys, and rejected records.

Batch, Micro-Batch, and Streaming Integration

Batch processing collects data and processes it at scheduled intervals. It is appropriate for payroll, daily inventory snapshots, periodic regulatory reports, and workloads in which minutes or hours of latency are acceptable. Its advantages include simpler reasoning, efficient bulk processing, and relatively straightforward recovery. Its weaknesses include delayed availability and the risk that a failure affects a large interval of data.

Micro-batch processing handles small groups of records at frequent intervals. It can provide near-real-time results while retaining some operational simplicity. Streaming processing treats records as continuously arriving events. A stream processor may filter, enrich, aggregate, and route events as they arrive. Streaming is useful for fraud detection, equipment monitoring, clickstream analysis, and alerting, but it introduces more complex concerns: event ordering, replay, state management, back-pressure, late arrival, and exactly-once or effectively-once behaviour.

Event time is the time at which an event occurred, whereas processing time is the time at which the system handled it. A network delay can cause these to differ. Windowed calculations should therefore define whether they use event time and how long the system waits for late events. A watermark is an operational estimate indicating that events older than a particular point are unlikely to arrive. It is a practical compromise: waiting longer improves completeness but delays results.

Delivery semantics require careful interpretation. At-most-once delivery may lose records but avoids duplicates. At-least-once delivery can retry records but may produce duplicates. Exactly-once processing attempts to ensure that each event affects the result once, but its guarantee usually depends on the entire source-to-sink design, not on a single messaging component. Idempotent writes, stable event identifiers, deduplication, and transactional or atomic sinks are often more realistic controls than assuming that a platform label solves the problem.

Sources of Integrated Data

Databases and Change Data Capture

A full extract reads an entire source table and is simple to understand, but it can be expensive and may place load on operational systems. An incremental extract reads only records changed since a stored checkpoint. It may use an updated timestamp, an increasing key, a version number, triggers, or a transaction log. Timestamp-based extraction is easy to implement but can miss updates caused by clock differences, unchanged timestamps, or late commits.

Change data capture reads changes made to a source, often from database transaction logs. A CDC record commonly includes an operation type such as insert, update, or delete, a source position, a transaction identifier, a key, and before-and-after values or the changed columns. CDC preserves change history and supports low-latency replication, but it requires careful handling of deletes, schema changes, log retention, transaction ordering, and source permissions.

A robust CDC design stores a durable checkpoint and makes application to the target idempotent. It should also distinguish a deleted record from a record that was never received. In analytical systems, deletions may be represented physically, through tombstone events, or logically using a deletion flag and validity period. The choice must align with legal retention requirements and analytical needs.

APIs and Files

APIs expose data through defined requests and responses. Integration designers must consider authentication, authorisation, pagination, rate limits, retries, timeouts, versioning, response schemas, and error codes. A retry policy should use exponential back-off and a limit; blindly retrying a failed request can overload a service or duplicate a non-idempotent operation. API ingestion should record request parameters, response status, retrieval time, and source version.

Files remain important because organisations exchange CSV, JSON, XML, spreadsheets, and fixed-width files. File pipelines should validate encoding, delimiters, headers, column counts, naming conventions, checksums, compression, and duplicate delivery. A file should not be treated as complete merely because it exists in a directory. Producers can use a temporary name and publish an atomic completion marker, allowing consumers to process only completed files.

Messaging and Event Streams

A message broker decouples producers and consumers. Topics or queues can distribute events to multiple applications, and partitions can support parallel processing. A message should contain a meaningful event identifier, event type, occurrence time, producer, schema version, entity key, and payload. Consumers should be able to acknowledge, retry, quarantine, and replay messages. A dead-letter queue or quarantine stream isolates records that cannot be processed without blocking all valid traffic.

Messaging improves resilience and scalability, but it does not automatically create data quality. An event with a valid JSON structure can still have an impossible quantity, an unknown customer, or an ambiguous time zone. Structural validity and business validity must be tested separately.

Transformation, Standardisation, and Master Data

Transformation changes the shape or meaning of data. Common operations include projection, filtering, joining, aggregation, parsing, unit conversion, code mapping, and derivation of business measures. Transformations should be deterministic where possible and should distinguish source fields from derived fields. Business rules such as “net sales exclude cancelled orders” must be documented with their owner and effective date.

Entity resolution is the process of determining when records refer to the same real-world entity. “A. Rahman,” “Abdul Rahman,” and an email address may identify one person, but matching should not rely on a single imperfect field. Deterministic rules, probabilistic matching, survivorship rules, and human review may be combined. False matches can merge distinct people, while missed matches can fragment a customer’s history.

Master data management (MDM) establishes trusted representations of important entities such as customers, products, suppliers, locations, and employees. MDM usually includes identifiers, definitions, stewardship, hierarchies, matching rules, and a process for resolving conflicts. A master record is not necessarily the most recent source record; it is the governed representation selected according to agreed policies. Reference data, such as country codes, currency codes, and product categories, provides controlled values used across systems.

A canonical model can simplify integration by providing a shared representation, but excessive standardisation can hide source-specific meanings or produce a model too complex for any team to maintain. A federated approach may be more appropriate when domains retain ownership while publishing governed data products.

Metadata, Lineage, and Data Contracts

Metadata is data about data. Technical metadata describes tables, columns, types, partitions, files, schemas, and pipeline dependencies. Business metadata explains definitions, owners, classifications, and permissible uses. Operational metadata records run times, row counts, statuses, latency, error rates, and checkpoints. Effective metadata enables discovery, impact analysis, quality monitoring, and compliance.

Data lineage records the origin and movement of data. It may be captured at the dataset, column, or transformation level. A lineage graph can answer questions such as: Which source fields contributed to a regulatory report? Which dashboards will be affected if a column changes? Which downstream products depend on a failing source? Lineage is most valuable when generated or maintained as part of pipeline deployment rather than documented once in a static diagram.

A data contract makes expectations explicit between producers and consumers. It may specify field names, types, allowed values, nullability, semantic definitions, freshness, volume expectations, privacy classification, and compatibility rules. Contracts do not eliminate change; they make change visible and manageable. A producer should deprecate a field through a communication and migration period rather than silently altering its meaning.

Data Quality as a Measurable Property

Data quality is fitness for a defined use, not an absolute characteristic. A value can be acceptable for exploratory analysis but unsuitable for a legal report. Quality assessment should therefore begin by identifying critical data elements, users, decisions, and tolerances.

Common dimensions include:

• Accuracy: correspondence with the real-world value or trusted source.
• Completeness: presence of required records or values.
• Validity: conformity to data types, formats, ranges, and domain rules.
• Consistency: absence of contradictory representations across records or systems.
• Uniqueness: absence of unintended duplicate entities or events.
• Timeliness: availability within the required time window.
• Integrity: preservation of relationships and constraints, such as valid foreign keys.
• Freshness: how recently a dataset or field was updated.

Metrics should be operational rather than decorative. If 9,800 of 10,000 required customer emails are non-null, completeness is 98 percent for that field, but this does not establish accuracy. A validity rule might require that order quantity be a positive integer; its pass rate is the number of records satisfying the rule divided by the number tested. A freshness metric may be the elapsed time since the latest accepted event. A reconciliation metric may compare source and target financial totals within a documented tolerance.

Quality rules can be classified as structural, content-based, referential, temporal, and business-rule checks. Structural checks verify schema and types. Content checks examine formats and ranges. Referential checks confirm that relationships resolve. Temporal checks detect future dates, impossible sequences, or stale records. Business checks test rules such as “a shipment cannot precede an order” or “a payment amount cannot be negative unless it is a refund.”

Not every failing record should stop a pipeline. A severity model can distinguish blocking failures, quarantinable errors, warnings, and accepted exceptions. Rejected records should be retained with reason codes, pipeline version, and source identifier so that they can be corrected and replayed. Silent dropping is one of the most damaging integration practices because it makes loss invisible.

Observability and Operational Control

Pipeline observability combines logs, metrics, traces, lineage, and alerts to explain what a system is doing and why. At minimum, an operational dashboard should show job status, duration, input and output counts, rejected records, freshness, throughput, lag, retry counts, and resource use. Data observability extends this view to schema changes, distribution shifts, null-rate changes, and unusual volumes.

Monitoring should focus on service-level objectives and business impact. A pipeline may complete successfully while delivering an empty dataset; technical success is not equivalent to data success. Useful alerts are actionable and routed to an accountable team. Alert fatigue results when every warning has the same priority or when thresholds are not connected to user impact.

Recovery design includes checkpointing, retry, replay, backfill, rollback, and quarantine. A failed batch should be restartable without creating duplicates. A streaming consumer should be able to resume from a known offset. Backfills should use a controlled code version and clearly identify affected partitions. Recovery procedures should be tested rather than assumed.

An Illustrative Case: Integrating Retail Orders for Near-Real-Time Analytics

Background

Consider a fictional retail organisation, Northstar Retail, operating online and physical stores. Online orders are stored in a relational database, store sales are uploaded as daily files, customer updates arrive through a CRM API, and payment and shipment services publish events. The organisation wants a unified sales dashboard, inventory alerts, and a customer analytics dataset.

Situation and Problem

The existing dashboard showed inconsistent sales totals. Online cancellations were sometimes counted as sales, store files were occasionally delivered twice, and customer identifiers differed between the CRM and order system. Managers also received inventory alerts several hours after stock movements. There was no shared definition of “net sale,” no record-level lineage, and no reliable procedure for correcting failed loads.

Analysis

The requirements were not identical. Financial and management reporting required reconciled daily data, while inventory alerts required low latency. A single batch architecture would not meet the alerting requirement, and a purely streaming architecture would complicate financial reconciliation. The main quality risks were duplicate files, missing cancellation events, entity mismatch, late shipment events, and unclear ownership of business definitions.

Approach and Solution

Northstar introduced a hybrid architecture. Database CDC captured order creation, updates, and cancellations. Store files were landed in an immutable area, checked for checksum and delivery identifier, and processed only after a completion marker appeared. The CRM API was extracted incrementally using a last-modified checkpoint and pagination. Payment and shipment events entered a broker with event identifiers and schema versions.

A raw zone preserved source records with ingestion metadata. Curated transformations standardised currency, time zones, product codes, and customer identifiers. A governed customer master maintained a cross-reference between source identifiers and a Northstar customer identifier. The business definition of net sales was documented as completed sales less valid cancellations and refunds, with effective dates for rule changes.

The analytical warehouse used idempotent merge operations. Each event was deduplicated by source and event identifier. Late events were accepted within a defined event-time window, after which a correction process recalculated affected daily aggregates. Failed records were quarantined with reason codes. Daily reconciliation compared order counts, cancellation counts, and monetary totals with source-system control totals.

A quality dashboard measured freshness, completeness of required keys, duplicate rate, referential integrity, schema compatibility, and reconciliation variance. Lineage connected dashboard measures to curated tables, transformations, and source fields. Data owners were assigned for orders, customers, products, and inventory.

Outcome

The architecture separated low-latency operational use from controlled reporting. More importantly, it made discrepancies explainable. A failed API extraction could be distinguished from a genuine fall in customer activity, and a duplicate file could be rejected without contaminating the warehouse. The case illustrates that integration improvement results from coordinated architecture, definitions, controls, and accountability rather than from selecting a single integration tool.

Learning Points

1. Different use cases justify different latency and processing models.

2. Stable identifiers and idempotent loading are essential when sources can retry or resend data.

3. Business definitions must be governed as carefully as schemas.

4. Quality metrics and lineage turn an opaque pipeline into an auditable service.

5. Hybrid architectures are often more practical than choosing ETL, ELT, batch, or streaming as exclusive alternatives.

Risks, Governance, and Professional Responsibility

Integration concentrates information and can amplify errors. A wrongly matched identity may affect credit, employment, healthcare, or access to services. Data minimisation, purpose limitation, retention controls, access management, encryption, masking, and audit logs are therefore integral to pipeline design. Sensitive fields should not be copied into every layer merely because storage is inexpensive.

Governance should assign data owners, stewards, producers, and consumers. Owners approve definitions and acceptable quality; stewards manage issues and metadata; engineering teams implement controls; consumers communicate use requirements. Access should follow least privilege, and service accounts should be managed separately from individual accounts. Privacy and regulatory obligations may require deletion, correction, consent tracking, residency controls, and evidence of processing.

Common mistakes include relying on full refreshes indefinitely, using timestamps without a safe overlap, ignoring deletes, assuming JSON validity means business validity, joining entities on weak identifiers, silently dropping rejected rows, changing schemas without notice, and treating a dashboard discrepancy as a reporting problem rather than a source-data problem. Another frequent error is optimising throughput while neglecting recoverability and auditability.

Automation, AI, and Emerging Developments

Cloud warehouses, lakehouses, workflow orchestrators, schema registries, data catalogues, and quality platforms increasingly automate ingestion, testing, lineage capture, and alerting. Infrastructure-as-code and version-controlled transformation models make environments reproducible. Automated profiling can identify changes in distributions, null rates, and cardinality.

Machine learning can assist entity resolution, anomaly detection, and classification of data-quality incidents. However, automated matching should expose confidence, preserve evidence, and provide a review path. A model that silently merges identities can create serious harm. Similarly, anomaly detection can identify unusual volumes but cannot determine whether a change is an error, a promotion, or a legitimate business event without contextual ownership.

Emerging practice includes data contracts enforced through schema registries, data mesh approaches that assign domain ownership, event-driven architectures, privacy-enhancing computation, and data products with published service-level expectations. These developments do not remove the need for relational reasoning, quality rules, and governance. They increase the importance of clear accountability because data is produced by many teams and consumed across many analytical and operational contexts.

Best Practices for Professional Implementation

A sound implementation begins with business use cases and critical data elements rather than with tool selection. Teams should document source ownership, expected latency, volume, security classification, definitions, and failure consequences. They should retain sufficient raw evidence for replay where lawful, use stable identifiers, and design every write for idempotence.

Schemas and data contracts should be versioned. Transformations and quality tests should be reviewed like application code. Pipelines should publish operational metadata, reconcile inputs and outputs, and quarantine rather than silently discard invalid records. Monitoring should cover both technical health and business quality. Recovery, backfill, and deletion procedures should be tested. Finally, quality incidents should be analysed for root causes and prevention, not merely corrected downstream.

Chapter Summary

Data integration makes heterogeneous data available as a coherent and governed resource. ETL transforms data before loading, whereas ELT loads data before applying transformations; hybrid architectures combine the strengths of both. Batch processing supports scheduled and high-volume work, while micro-batch and streaming support lower latency but require controls for ordering, duplication, late events, state, and replay.

Databases, CDC, APIs, files, and messaging systems each have distinct extraction and failure characteristics. Reliable pipelines use checkpoints, stable identifiers, idempotent writes, schema management, reconciliation, and recoverability. Transformation must address standardisation, entity resolution, master data, reference data, and documented business definitions. Metadata, lineage, and data contracts make data discoverable, traceable, and changeable without unnecessary disruption.

Data quality is fitness for use and should be measured through dimensions such as accuracy, completeness, validity, consistency, uniqueness, timeliness, integrity, and freshness. Observability combines technical and business monitoring, while governance addresses privacy, access, retention, accountability, and ethical risk. Modern automation and AI can improve profiling, anomaly detection, and matching, but they require transparency and human oversight. Effective integration is consequently a socio-technical discipline involving architecture, operations, management, and professional responsibility.

Key Terms

Data integration: The coordinated combination of data from multiple sources for shared use.

ETL: Extract, transform, and load; a pattern that transforms data before loading it into a target.

ELT: Extract, load, and transform; a pattern that transforms data within the target platform.

Batch processing: Processing data in scheduled groups rather than continuously.

Streaming: Continuous processing of arriving events, often with low latency.

Change data capture: A method of identifying and propagating inserts, updates, and deletes from a source.

Idempotence: The property that repeating an operation does not create unintended additional effects.

Schema drift: An unplanned or evolving change in data structure or meaning.

Data contract: An agreement specifying a dataset’s schema, meaning, quality, and change expectations.

Metadata: Information describing the structure, meaning, ownership, or operation of data.

Data lineage: A record of data origin, transformations, movement, and dependencies.

Master data management: Governance and control of authoritative representations of important entities.

Entity resolution: The process of deciding which records refer to the same real-world entity.

Data quality: The suitability of data for a defined purpose, assessed through measurable dimensions.

Watermark: A processing-time boundary used to manage the arrival of late events.

Dead-letter queue: A destination for messages that cannot be processed successfully.

Data observability: Monitoring that explains the health, freshness, structure, and behaviour of data systems.

Reference data: Controlled codes or values used consistently across systems.

Data product: A governed, discoverable dataset or data service with an owner and defined users.

Reconciliation: Comparison of source and target results to detect loss, duplication, or transformation error.

Review Questions

Short Answer Questions

1. What is the difference between ETL and ELT?

2. Why is idempotence important in data pipelines?

3. Distinguish event time from processing time.

4. What purposes do metadata and data lineage serve?

5. Name four dimensions of data quality and briefly explain each.

Long Answer Questions

1. Explain the architecture and control requirements of a reliable batch integration pipeline.

2. Compare batch, micro-batch, and streaming processing, including their operational challenges.

3. Describe how CDC works and discuss the treatment of updates, deletes, checkpoints, and schema changes.

4. Explain the roles of master data management, entity resolution, reference data, and data contracts.

5. Discuss how governance, observability, and data-quality management should be integrated into an enterprise pipeline.

Analytical / Application Questions

1. A daily customer file is occasionally delivered twice and sometimes contains records with changed identifiers. Design controls for ingestion, deduplication, entity resolution, reconciliation, and correction.

2. A fraud-detection stream receives events out of order and at least once. Propose a design using event identifiers, windows, watermarks, retries, and idempotent storage.

3. A regulatory report differs from the source system by 3 percent, although the pipeline reports successful completion. Analyse possible causes and develop an investigation and remediation plan using quality metrics, lineage, and observability.

22. Multiple Choice Questions with Answers

Question 1

Which situation most strongly supports an ELT architecture?

A. The target platform has scalable compute and raw data must be retained for later analysis

B. All data must be filtered before entering a highly restricted target

C. The source system cannot export data in batches

D. The organisation has no need for transformation logic

Correct Answer: A. The target platform has scalable compute and raw data must be retained for later analysis

Question 2

A pipeline receives the same payment event twice. Which control most directly prevents duplicate financial effects?

A. Increasing the number of processing partitions

B. Using an event identifier with an idempotent sink operation

C. Converting the event from JSON to CSV

D. Increasing the dashboard refresh frequency

Correct Answer: B. Using an event identifier with an idempotent sink operation

Question 3

What is the primary purpose of a CDC checkpoint?

A. To define a customer’s master record

B. To record the last safely processed source position

C. To encrypt the transaction log

D. To calculate data completeness

Correct Answer: B. To record the last safely processed source position

Question 4

Why can a technically successful pipeline still produce a serious data problem?

A. Successful jobs always process too slowly

B. A completed job may load empty, duplicated, or semantically incorrect data

C. Technical monitoring cannot record job duration

D. Streaming systems cannot use validation rules

Correct Answer: B. A completed job may load empty, duplicated, or semantically incorrect data

Question 5

Which quality dimension is most directly assessed by checking that every order references an existing customer?

A. Timeliness

B. Uniqueness

C. Referential integrity

D. Freshness

Correct Answer: C. Referential integrity

Question 6

In a streaming system, what problem does a watermark help address?

A. Encryption of sensitive fields

B. Management of events that arrive after their event-time window

C. Creation of a master-data hierarchy

D. Conversion of relational tables into files

Correct Answer: B. Management of events that arrive after their event-time window

Question 7

Which practice is safest when an API extraction fails temporarily?

A. Retry indefinitely without delay

B. Ignore the failure and publish an empty dataset

C. Use bounded retries with back-off and preserve the extraction checkpoint

D. Delete all previously extracted records

Correct Answer: C. Use bounded retries with back-off and preserve the extraction checkpoint

Question 8

What is the main value of data lineage for a regulatory report?

A. It increases network bandwidth

B. It shows which sources and transformations contributed to the report

C. It automatically guarantees accuracy

D. It replaces access control

Correct Answer: B. It shows which sources and transformations contributed to the report

Question 9

A store file appears in a landing directory before writing is complete. Which design best prevents partial ingestion?

A. Process every file immediately upon appearance

B. Require an atomic completion marker or temporary-to-final rename

C. Ignore file metadata and infer completion from file size once

D. Convert the file to a different delimiter

Correct Answer: B. Require an atomic completion marker or temporary-to-final rename

Question 10

Which statement best describes data quality?

A. It is an absolute property independent of use

B. It concerns only whether values are non-null

C. It is fitness for a defined purpose measured through relevant dimensions

D. It is guaranteed when a database has primary keys

Correct Answer: C. It is fitness for a defined purpose measured through relevant dimensions