4. Data Integration, ETL and Data Quality

Introduction and Context

A data warehouse is valuable only when the data stored in it is sufficiently complete, consistent, timely, understandable, and trustworthy. In practice, analytical systems rarely receive data from a single, uniform source. They draw from enterprise resource planning systems, customer relationship management platforms, operational databases, application programming interfaces, spreadsheets, log streams, external reference datasets, and increasingly, cloud services and Internet-of-Things devices. These sources differ in structure, naming conventions, update frequency, granularity, encoding, security controls, and degrees of reliability.

Data integration provides the technical and organizational means for bringing such heterogeneous data together. Extract, transform, and load (ETL) is the traditional process used to move data from source systems into a target warehouse or data mart. Extract, load, and transform (ELT) reverses the order of the last two stages and has become prominent in cloud data platforms, where large-scale computation can be performed after data has been loaded into the target environment.

Neither ETL nor ELT is merely a file-transfer activity. It involves understanding source systems, interpreting business meaning, identifying and correcting quality problems, matching records that refer to the same entity, detecting changes, preserving lineage, validating results, and coordinating dependencies. A pipeline that loads data quickly but produces duplicated customers, inconsistent currencies, or incomplete transactions can damage decision-making more seriously than a pipeline that fails visibly.

This chapter examines the architecture and practice of data integration as a controlled engineering discipline. It links technical mechanisms with data governance, quality measurement, operational monitoring, and managerial accountability.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the role of data integration, ETL, ELT, staging, and data pipelines in analytical architectures.

2. Profile source systems and identify structural, semantic, and quality problems before extraction.

3. Design extraction, cleansing, transformation, matching, deduplication, change-data-capture, and loading procedures.

4. Distinguish batch, incremental, streaming, ETL, and ELT approaches and select among them appropriately.

5. Define data-quality dimensions, measurements, validation rules, and data-quality service levels.

6. Use metadata, lineage, orchestration, audit controls, and reconciliation to make pipelines reliable.

7. Evaluate risks involving privacy, security, governance, bias, operational failure, and inappropriate automation.

8. Apply these ideas to the design and diagnosis of an enterprise data-integration solution.

The Integration Problem: More Than Moving Data

Data integration creates a coherent view from data that was produced for different operational purposes. An order-management system may store one row per order line, while a finance system stores one row per invoice and a marketing system stores one row per campaign interaction. A warehouse must preserve the distinctions among these events while also making it possible to analyze them together.

Three forms of heterogeneity are especially important. Structural heterogeneity occurs when systems use different schemas, data types, keys, or levels of detail. Syntactic heterogeneity concerns representation: dates may appear as `2025-04-01`, `01/04/2025`, or a locale-specific text value. Semantic heterogeneity occurs when apparently similar fields have different meanings. For example, “customer” may mean a billing account in one system and an individual contact in another.

Integration therefore requires a semantic agreement about entities, attributes, events, measures, time, and relationships. A common data model or canonical model can help, but it should not erase distinctions that matter analytically. A successful integration architecture preserves source evidence while presenting consistent, documented business interpretations.

Important Terminology

A source system is an application, database, file, service, or device that provides data. A target system is the warehouse, lakehouse, data mart, operational data store, or other destination that receives integrated data. A staging area is an intermediate location in which extracted data is temporarily stored and examined before further processing.

A data pipeline is the sequence of technical and logical operations through which data is acquired, processed, and delivered. A data flow describes movement and transformation of data; a control flow describes dependencies, decisions, retries, and execution order. Metadata is data about data, including definitions, schemas, ownership, quality results, and processing history.

A business key has meaning in a business domain, such as an account number. A surrogate key is an internally generated identifier used in an integrated model. A conformed dimension has consistent meaning and attributes across subject areas. Granularity, or grain, specifies what one record represents, such as one product per warehouse per day or one invoice line.

Profiling Sources Before Designing the Pipeline

Source profiling is the systematic examination of data before integration rules are finalized. Profiling should be performed at both technical and business levels. Technical profiling examines data types, lengths, null rates, value distributions, uniqueness, patterns, referential relationships, and update timestamps. Business profiling asks what each field means, who owns it, when it becomes valid, and how it should be used.

Typical profiling questions include:

• What entities and events are represented?
• What is the grain of each table or file?
• Which columns form candidate keys?
• Are values complete, unique, valid, and consistent?
• Which fields change, and how frequently?
• Are dates recorded in a common time zone?
• Are codes documented and stable?
• What records are deleted or corrected in the source?
• Does the source distinguish unknown, not applicable, and missing values?

A profiling report should record statistics as well as exceptions. For example, reporting that 98 percent of postal codes match an expected pattern is more useful than simply labeling the column “valid.” Profiling can also reveal data drift: a source column previously containing numeric identifiers may begin to contain alphanumeric values after an application release.

Data profiling is not a substitute for domain knowledge. A column with unique values may appear to be a key but may actually be a transaction sequence that is unique only within a branch. Similarly, a non-null field may contain placeholder values such as “N/A,” “unknown,” or zero. Profiling results should therefore be reviewed with data owners and subject-matter experts.

Extraction: Acquiring Source Data Safely

Extraction obtains data from source systems while minimizing disruption and preserving sufficient evidence for later processing. A full extraction reads all relevant source records. It is straightforward but can be expensive and may place excessive load on operational systems. An incremental extraction obtains only new or changed records since a previous successful extraction.

Incremental extraction may use a reliable last-modified timestamp, an increasing sequence number, database logs, triggers, source-issued change tables, or application programming interfaces that expose change tokens. The extraction design must answer two questions: how are changes identified, and how are deletions detected? A timestamp-only approach can miss updates when clocks are inaccurate, timestamps have low precision, or records are modified without updating the field. Deletions are particularly difficult unless the source provides tombstones, audit logs, snapshots for comparison, or a deletion flag.

Change-data capture (CDC) records inserts, updates, and deletes from a source. Log-based CDC is generally less intrusive than triggers because it reads database transaction logs, but it requires suitable database support and careful handling of ordering, transactions, schema changes, and recovery. CDC pipelines must be idempotent: replaying the same change should not create an unintended additional effect.

Extraction should use controlled connections, least-privilege credentials, encryption in transit, rate limits, and checkpointing. A checkpoint should be advanced only after the extracted data has been durably recorded or successfully processed according to the pipeline design. Otherwise, a failure between reading and recording the checkpoint can cause data loss.

Cleansing and Standardization

Cleansing improves fitness for intended use; it does not mean arbitrarily changing source values. Common operations include trimming whitespace, standardizing case, parsing dates, validating domains, normalizing units, correcting known code mappings, handling invalid characters, and separating compound fields.

Null handling requires semantic care. A missing telephone number, a telephone number that was intentionally withheld, and a telephone number not applicable to an organization are not necessarily equivalent. A warehouse may use explicit status indicators or standardized unknown members rather than silently converting every missing value to an empty string.

Standardization should be governed by documented rules. For example, address normalization may use postal reference data, but an automated correction should retain the original value and record the rule applied. Unit conversion must preserve precision and identify the source unit. Currency conversion requires an exchange-rate policy, an effective date, and a decision about whether historical values are restated.

Cleansing can produce exceptions rather than forced corrections. Records that fail a critical validation may be placed in a quarantine area with an error code, source identifier, processing timestamp, and remediation status. This prevents bad records from contaminating the target while ensuring that data-quality failures are visible and actionable.

Transformation and Semantic Integration

Transformation converts source representations into target structures and business meanings. It may include joins, filtering, aggregation, derivation of measures, code translation, calendar mapping, hierarchy construction, and application of business rules.

A transformation specification should identify the source fields, target fields, rule logic, data types, treatment of nulls, effective dates, and ownership. Consider revenue: one system may record gross order value, another net invoiced value, and a third recognized revenue. A target measure named “revenue” is unsafe unless its definition, grain, timing, and exclusions are explicit.

Dimensional models often use surrogate keys to isolate the warehouse from changes in source identifiers. When a fact record arrives, the pipeline looks up the appropriate dimension version and assigns its surrogate key. Slowly changing dimensions manage changes in descriptive attributes. In a Type 1 approach, the old value is overwritten; in Type 2, a new version is inserted with effective and expiry dates; other approaches preserve limited history. The choice depends on whether historical analysis requires the former value.

Transformation should be deterministic where possible. If the same inputs and reference data are processed twice, the expected output should be the same. Reference data, rule versions, and exchange rates must therefore be controlled and auditable.

Entity Resolution and Deduplication

Integration frequently encounters multiple records for the same real-world entity. Entity resolution, also called record linkage, determines which records refer to the same person, organization, product, or location. Deduplication removes or consolidates redundant representations after that determination.

A deterministic method uses exact or highly reliable keys, such as a validated customer identifier. Probabilistic or fuzzy matching evaluates similarities in names, addresses, telephone numbers, email addresses, and other attributes. Similarity algorithms may use edit distance, token comparison, phonetic encoding, or domain-specific rules. A practical design normally defines three outcomes: automatic match, automatic non-match, and review required.

Matching thresholds must be evaluated against the costs of false positives and false negatives. Merging two different people can be more harmful than leaving two records temporarily separate, particularly in credit, healthcare, or regulatory contexts. Match decisions should be explainable and stored with evidence, scores, rules, and reviewer actions.

A survivorship policy determines which attribute values are retained when records are merged. The most recent value is not always the most reliable; a verified address from a trusted source may take precedence over a newer unverified value. Golden records should remain traceable to contributing source records rather than becoming opaque replacements.

Loading Strategies and Warehouse Consistency

Loading places prepared data into the target environment. Full refresh loading replaces a target table or partition and is useful when the dataset is small or reproducibility is more important than minimal processing. Incremental loading applies only new or changed data and is generally more efficient for large systems.

A robust load uses staging tables, batch identifiers, transaction boundaries where supported, constraints or post-load checks, and an audit record. It should define what happens when a batch partially fails. Options include rollback, restart from a checkpoint, correction and replay, or acceptance of a partial batch with an explicit status. Silent partial loading is unacceptable for trusted analytical systems.

Fact loading must respect referential dependencies. Dimensions or reference entities may need to be loaded before facts. Late-arriving dimensions and facts require special handling, such as an inferred member that is updated when full descriptive data arrives. Partitioning, bulk loading, parallel processing, and columnar storage can improve performance, but optimization must not undermine correctness or auditability.

ETL performs substantial transformation before loading, which can reduce target-storage demands and enforce centralized rules. ELT loads relatively raw data first and uses the target platform for transformation, increasing flexibility and preserving source detail. ETL is often suitable when the target has limited compute or when sensitive data must be filtered before arrival. ELT is attractive when elastic cloud compute, reusable raw data, and rapid experimentation are priorities. Hybrid architectures are common: sensitive fields may be masked during extraction, raw data retained in a controlled zone, and business transformations performed later.

Orchestration, Lineage, and Observability

A production pipeline requires orchestration rather than a collection of unrelated scripts. An orchestrator schedules tasks, expresses dependencies, manages retries, records states, and raises alerts. It should distinguish transient failures, such as temporary network unavailability, from permanent data errors, such as an invalid schema. Retries must be bounded and designed to avoid duplicate effects.

Pipeline observability includes logs, metrics, traces, freshness indicators, record counts, processing duration, rejected-record counts, and source-to-target reconciliation. A dashboard should make it possible to answer whether a pipeline ran, whether it processed the expected data, whether quality changed, and whether users received the data on time.

Data lineage documents where data originated, how it was transformed, and where it is consumed. Technical lineage may connect source columns to target columns and transformation jobs. Business lineage explains how a report measure is defined and which organizational owner is accountable. Lineage supports impact analysis, debugging, regulatory response, and trust. It is most useful when captured automatically from pipeline metadata and supplemented with business definitions.

Data Quality as a Measurable Property

Data quality is fitness for a defined use, not an abstract property that is identical for every dataset. A marketing dataset may tolerate missing phone numbers but not missing consent status. A financial report may require strict completeness and reconciliation, while an exploratory dataset may accept provisional values if they are clearly labeled.

Important dimensions include:

• Completeness: the extent to which required values or records are present.
• Validity: conformity to permitted formats, domains, types, or rules.
• Accuracy: closeness to the correct real-world value.
• Consistency: absence of contradictions within or across datasets.
• Uniqueness: absence of unintended duplicate records.
• Timeliness: availability and currency within the required period.
• Integrity: preservation of relationships and referential constraints.

Metrics should be expressed operationally. A completeness ratio might be calculated as the number of non-null required values divided by the number of applicable records. A duplicate rate might be the number of records belonging to duplicate groups divided by total records. A validity metric may count values passing a rule, while a freshness metric measures the time between source creation and target availability.

Quality rules require thresholds and owners. For example: “At least 99.5 percent of active customer records must contain a valid country code; violations above the threshold create a warning, while violations above a higher threshold stop publication.” Thresholds should be risk-based rather than arbitrary. Trends are often more informative than one isolated score: a gradual deterioration may reveal an upstream application change before users report a problem.

Quality management includes prevention, detection, correction, and monitoring. Prevention may involve input controls and reference data. Detection occurs through profiling and validation. Correction may be performed at the source, in a governed integration layer, or by an accountable data steward. Monitoring ensures that corrections remain effective.

Validation and Reconciliation

Validation operates at several levels. Schema validation checks columns, types, and required structures. Record-level validation checks individual values. Cross-record validation checks uniqueness and patterns. Cross-table validation checks relationships. Business validation checks rules such as invoice totals equaling the sum of approved line items. Reconciliation compares source and target counts, sums, control totals, and change populations.

A count match alone does not prove correctness. A missing record and an extra record can produce equal counts. Reconciliation should therefore combine counts with totals, key-range checks, hash totals where appropriate, and sampled or complete key comparisons. Differences should be classified as expected transformation effects, timing differences, source defects, or pipeline defects.

Testing should include unit tests for transformation logic, integration tests across components, regression tests for previously corrected defects, performance tests, and failure-recovery tests. Data contracts can specify the schema, semantics, quality expectations, and change-notification obligations between producers and consumers.

An Illustrative Case: Integrating a Multi-Channel Retailer

Background

Assume that Northstar Retail, a fictional national retailer, operates physical stores, an e-commerce platform, a loyalty application, and a finance system. Management wants a unified view of customer value, inventory movement, returns, and promotional effectiveness.

Situation and Problem

The online system identifies customers by email address, stores use loyalty-card numbers, and finance uses account identifiers. Names and addresses vary across systems. Online orders are updated several times before shipment, while store transactions arrive in nightly files. Returns may be recorded days after the original sale. Executives have observed that customer counts differ among reports and that daily revenue totals do not reconcile.

Analysis

Initial profiling finds that approximately some records—without assuming a precise enterprise-wide percentage—contain blank or placeholder emails, store files use multiple date formats, and product codes are not stable across an acquired subsidiary. The online system exposes update timestamps but no reliable deletion feed. Store files contain both sales and returns but do not explicitly identify every original transaction. A key semantic issue is that finance recognizes revenue after approval, whereas operational sales report the transaction at purchase time.

The integration team defines separate grains: one row per online order line, one row per store transaction line, one row per finance invoice line, and one row per return line. It refuses to combine these into a single ambiguous “sales” table. Business owners approve distinct measures for gross sales, approved revenue, discounts, and returns.

Approach and Solution

The team establishes a raw landing zone with immutable, encrypted extracts and batch identifiers. Online changes are captured using a timestamp plus an overlap window; records are re-read for a short period so that late-arriving updates are less likely to be missed. A comparison process identifies deletions through periodic snapshots. Store files are loaded into staging and validated against file-control totals.

Product codes are mapped through a governed cross-reference table. Customer matching first uses validated loyalty identifiers, then exact normalized email where consent and reliability conditions permit, and finally a scored address-and-name process. Borderline cases enter a steward review queue. The resulting customer dimension uses surrogate keys and Type 2 history for selected attributes, while retaining source identifiers and match evidence.

Data-quality rules check date validity, product mapping, nonnegative quantities except for explicitly coded returns, and reconciliation of financial control totals. Failed records are quarantined rather than silently discarded. An orchestrated workflow loads dimensions before facts, records lineage from each report measure to its source transformations, and publishes a quality status with each daily dataset.

Outcome

The retailer obtains separate but reconcilable views of operational sales and recognized revenue. Duplicate customer rates decline after stewardship review, and report users can distinguish a late source file from a genuine business decline. Most importantly, disagreements are no longer resolved by manually editing reports; they are traced to definitions, source records, or documented pipeline exceptions.

Learning Points

The case demonstrates that integration begins with grain and semantics, not with tool selection. Matching requires risk-sensitive controls. Reconciliation must reflect business meaning rather than rely only on row counts. Finally, quality, lineage, and operational status are part of the product delivered to users.

Risks, Governance, and Professional Responsibilities

Poor integration can propagate errors at scale. Common mistakes include loading before profiling, treating null as zero, using unstable business keys as permanent identifiers, ignoring deletions, applying fuzzy matching without review, overwriting history, and allowing failed records to disappear. Another frequent error is optimizing for pipeline speed while neglecting recoverability and audit evidence.

Privacy and security must be designed into the pipeline. Personal data should be collected for legitimate purposes, minimized where possible, encrypted in transit and at rest, and exposed according to role and purpose. Masking, tokenization, access logging, retention limits, and secure deletion may be required. Data lineage should identify sensitive fields and downstream consumers. Consent and purpose limitations can affect whether records may be matched or reused across domains.

Governance assigns accountability. Data owners approve definitions and quality thresholds; data stewards investigate and remediate issues; engineers implement controls; security and privacy specialists assess protection; business users validate usefulness. Automated data correction and AI-assisted matching should be explainable, monitored for unequal error rates, and subject to human review in high-impact decisions.

Current Developments and Best Practices

Cloud warehouses and lakehouses increasingly support ELT, open table formats, scalable processing, and near-real-time ingestion. Data virtualization and federation can provide access without physically copying every dataset, although they do not remove semantic or quality problems. Data observability tools use automated monitors to detect freshness, volume, schema, and distribution anomalies. Data contracts and data-product thinking place clearer quality obligations on producing teams.

Machine learning can assist entity resolution, anomaly detection, schema matching, and rule discovery. It should augment—not replace—domain governance. Models require labeled examples, threshold calibration, drift monitoring, and explanations of match or anomaly decisions.

The following practices provide a reliable foundation:

1. Define business meaning and grain before writing transformations.

2. Profile sources and maintain documented data contracts.

3. Preserve raw evidence while separating it from curated data.

4. Make pipelines incremental, restartable, idempotent, and observable.

5. Treat deletions, late-arriving data, and schema changes explicitly.

6. Store lineage, rule versions, batch identifiers, and rejected-record reasons.

7. Use risk-based quality thresholds with named owners and escalation paths.

8. Reconcile both technical totals and business measures.

9. Test failure recovery as seriously as successful execution.

10. Protect personal and confidential data throughout its lifecycle.

Chapter Summary

Data integration converts heterogeneous operational data into coherent, usable analytical data. ETL transforms data before loading, whereas ELT loads data earlier and uses target-platform computation; hybrid designs combine both. Reliable integration begins with source profiling and semantic clarification, followed by controlled extraction, cleansing, transformation, entity resolution, deduplication, change-data capture, and loading. Incremental processing must address updates, deletions, late-arriving data, checkpoints, and idempotence.

Production pipelines depend on orchestration, metadata, lineage, monitoring, reconciliation, and failure recovery. Data quality is fitness for a defined purpose and should be measured through dimensions such as completeness, validity, accuracy, consistency, uniqueness, timeliness, and integrity. Governance assigns ownership and protects privacy, while emerging cloud, observability, data-contract, and AI capabilities extend integration practice without eliminating the need for human judgment.

Key Terms

Data integration: The process of combining data from heterogeneous sources into a coherent environment or view.

ETL: Extract, transform, and load; a pipeline pattern in which transformation occurs before target loading.

ELT: Extract, load, and transform; a pattern in which data is loaded before transformation in the target environment.

Staging area: An intermediate storage location used to hold and inspect extracted data.

Data profiling: Systematic analysis of data structure, content, patterns, and quality characteristics.

Change-data capture: A method for identifying inserts, updates, and deletes occurring in a source system.

Idempotence: The property that repeated processing of the same input produces no unintended additional effect.

Data lineage: Documentation of data origins, transformations, movement, and downstream use.

Metadata: Information describing data, its structure, meaning, ownership, quality, and processing history.

Grain: The precise business meaning represented by one record in a dataset.

Business key: An identifier meaningful within a business process or domain.

Surrogate key: An internally generated identifier used by an integrated data model.

Entity resolution: The process of determining whether records represent the same real-world entity.

Deduplication: Identification and treatment of redundant representations of an entity.

Data contract: An agreement specifying a dataset’s schema, semantics, quality, and change obligations.

Completeness: The extent to which required records or values are present.

Validity: Conformity of data to specified formats, types, domains, or rules.

Reconciliation: Comparison of source and target data to verify completeness and correctness of transfer.

Orchestration: Coordination of pipeline tasks, dependencies, schedules, retries, and execution states.

Slowly changing dimension: A dimensional-modeling technique for managing changes in descriptive attributes over time.

Review Questions

Short Answer Questions

1. Distinguish structural heterogeneity from semantic heterogeneity.

2. Why is source profiling necessary before transformation design?

3. What problem does change-data capture address?

4. Define data lineage and state two of its uses.

5. Why should a data-quality rule specify an owner and threshold?

Long Answer Questions

1. Explain the complete lifecycle of an ETL pipeline from source profiling through validation and publication.

2. Compare ETL and ELT with respect to performance, governance, flexibility, security, and cloud implementation.

3. Discuss deterministic and probabilistic entity resolution, including their risks and governance requirements.

4. Explain how orchestration, checkpointing, idempotence, and reconciliation contribute to reliable pipelines.

5. Describe the major dimensions of data quality and propose suitable metrics for an enterprise warehouse.

Analytical / Application Questions

1. A source system provides a last-modified timestamp but no deletion indicator. Design an incremental extraction and deletion-detection strategy, identifying its assumptions and risks.

2. Two customer records have similar names and addresses but different government identifiers. Explain whether they should be merged automatically, routed for review, or kept separate, and justify your decision.

3. A daily sales load has matching row counts but a substantial difference in total value from the source. Diagnose possible causes and propose validation and remediation steps.

22. Multiple Choice Questions with Answers

Question 1

Which situation most clearly represents semantic heterogeneity?

A. One system stores dates as strings and another stores them as date types.

B. One system defines a customer as a billing account while another defines it as an individual contact.

C. Two systems use different column names for the same identifier.

D. One source is stored in a comma-separated file and another in a database.

Correct Answer: B. One system defines a customer as a billing account while another defines it as an individual contact.

Question 2

Why should a CDC pipeline be idempotent?

A. To ensure that all source records are always processed in alphabetical order.

B. To allow repeated or replayed changes without creating unintended duplicate effects.

C. To eliminate the need for source-system profiling.

D. To guarantee that source data contains no null values.

Correct Answer: B. To allow repeated or replayed changes without creating unintended duplicate effects.

Question 3

Which extraction problem is especially difficult when a source provides no deletion indicator?

A. Detecting records that have been removed from the source.

B. Converting a source date into a target date.

C. Calculating a target surrogate key.

D. Loading a reference table before a fact table.

Correct Answer: A. Detecting records that have been removed from the source.

Question 4

A retailer has two customer records with similar names but different validated identifiers. What is the most appropriate general response?

A. Merge them automatically because name similarity is sufficient.

B. Delete both records to avoid duplication.

C. Avoid automatic merging and use identifiers or governed review to resolve the conflict.

D. Replace both names with a single standardized value.

Correct Answer: C. Avoid automatic merging and use identifiers or governed review to resolve the conflict.

Question 5

Which metric is most directly associated with timeliness?

A. The percentage of required fields that are non-null.

B. The percentage of records conforming to an allowed code list.

C. The elapsed time between source creation and target availability.

D. The proportion of records belonging to duplicate groups.

Correct Answer: C. The elapsed time between source creation and target availability.

Question 6

Why is a row-count match insufficient for source-to-target reconciliation?

A. Rows cannot be counted in a staging area.

B. An extra record and a missing record can offset one another while counts remain equal.

C. Row counts are useful only for streaming pipelines.

D. Row counts measure only data validity and never volume.

Correct Answer: B. An extra record and a missing record can offset one another while counts remain equal.

Question 7

Which characteristic best distinguishes ELT from traditional ETL?

A. ELT transforms data before it leaves the source system.

B. ELT loads data into the target platform before performing substantial transformation there.

C. ELT cannot support incremental processing.

D. ELT eliminates the need for metadata and lineage.

Correct Answer: B. ELT loads data into the target platform before performing substantial transformation there.

Question 8

A pipeline fails after loading data but before recording its checkpoint. Which design property most helps safe recovery?

A. Idempotence combined with durable staging or batch tracking.

B. Replacing all nulls with zeros.

C. Removing audit logs to reduce storage.

D. Using only fuzzy matching for source records.

Correct Answer: A. Idempotence combined with durable staging or batch tracking.

Question 9

Which practice best supports explainable entity resolution?

A. Keeping only the final golden record and deleting source evidence.

B. Recording match rules, scores, evidence, and reviewer decisions.

C. Using the newest value regardless of source reliability.

D. Applying a single threshold without monitoring false matches.

Correct Answer: B. Recording match rules, scores, evidence, and reviewer decisions.

Question 10

A data-quality rule states that 99.5% of active customer records must have valid country codes, with a named steward and escalation process. What does this illustrate?

A. A data-quality service-level expectation.

B. A surrogate-key generation method.

C. A physical database partition.

D. A source deletion algorithm.

Correct Answer: A. A data-quality service-level expectation.